Code Monkey home page Code Monkey logo

webav's Introduction

WebAV

An audio/video toolkit built with pure web technologies, planned to include creation, editing, and exporting capabilities, applicable to products like live streaming, tutorial recording, video clipping, etc.

使用纯 Web 技术构建的音视频工具,计划包含创建、编辑、导出功能,可应用于直播、教程录制、视频剪辑等产品中。

Motivation

Chrome 94 released the WebCodecs API, meaning JS can now handle audio/video as well. Before this, frontend developers could only use ffmpeg.wasm in limited scenarios. Through simple tests of decoding and re-encoding videos, it was found WebCodecs has 20x the performance of ffmpeg.wasm.

Chrome 94 开放了 WebCodecs API,意味着 JS 也能处理音视频了。在此之前,前端开发在会在有限的场景使用 ffmpeg.wasm,经过简单地测试解码和重编码视频发现,WebCodecs 性能是 ffmpeg.wasm 的 20 倍。

This is an experimental project attempting to provide easy-to-use APIs for handling audio/video data in the browser. The project is under active development, feel free to open issues to report bugs or suggest new features.

这是一个实验性项目,尝试提供简单易用的 API 在浏览器中处理音视频数据。项目正在积极迭代,可提交 issue 来反馈 Bug 和新功能建议。

Packages

  • AVCliper
    Audio/Video Clipping Toolkit.
    音视频剪辑工具库。
  • AVCanvas
    Combine Text, Image, Video, Audio, UserMedia, DisplayMedia to generate MediaStream.
    使用文字、图片、音视频文件、摄像头&麦克风、分享屏幕来生成 MediaStream。
  • AVRecorder
    Record MediaStream, export MP4 stream.
    录制 MediaStream,导出 MP4 流。

Features

  • Audio/Video Clipper AVCliper

    • Combine mp4, mp3, images, text by timeline
    • Embed SRT subtitles
    • Video muting, audio mixing, volume control, audio looping, audio resampling
    • Video frame by frame processing, built-in chroma key
    • Animation properties: x, y, w, h, opacity, angle
  • Canvas AVCanvas

    • Move, rotate, flip, scale to fit, warp
    • Media sources: camera, microphone, screen sharing
    • Resource types: video, audio, image, text
  • Recorder AVRecorder

    • Record video from MediaStream, AVCanvas, export to MP4
  • 音视频剪辑 AVCliper

    • mp4、mp3、图片、文字 按时间线合成
    • 嵌入 SRT 字幕
    • 视频消音、音频混流、音量控制、音频循环、音频重采样
    • 视频逐帧处理,内置 绿幕抠图
    • 动画属性:x, y, w, h, opacity, angle
  • 画布 AVCanvas

    • 移动、旋转、翻转、等比缩放、变形
    • 媒体来源:摄像头、麦克风、分享屏幕
    • 资源类型:视频、音频、图片、文字
  • 录制 AVRecorder

    • 从 MediaStream、AVCanvas 中录制视频,导出 MP4

DEMO

https://hughfenghen.github.io/WebAV/demo/

Development

This repo uses lerna + yarn to manage packages.

  1. Install dependencies: Run yarn install in root directory
  2. Build all packages: Run yarn build in root directory
  3. Run DEMO
    1. cd into specific package, e.g. cd packages/av-cliper
    2. Run yarn dev
  4. Access DEMO in browser
    1. Copy baseUrl printed in console, e.g. http://localhost:6066/
    2. Locate DEMO path, it's filename in packages/av-cliper/demo dir, like concat-media.html
    3. Open final URL in browser: http://localhost:6066/concat-media.html

当前仓库是使用 lerna + yarn 管理 packages。

  1. 安装依赖:在根目录执行 yarn install
  2. 构建所有 packages:在根目录执行 yarn build
  3. 运行 DEMO
    1. cd 到特定的 package 目录,比如 cd packages/av-cliper
    2. 执行 yarn dev
  4. 访问 DEMO
    1. 复制控制台中打印的 baseUrl,类似 http://localhost:6066/
    2. 确定对应 DEMO 的 path,是 packages/av-cliper/demo 目录下的文件名,如 concat-media.html
    3. 在浏览器打开最终 URL:http://localhost:6066/concat-media.html

若你有意贡献代码,建议先跟通过 Github Issue、Discussions、微信(liujun_fenghen)与作者讨论


加微信 liujun_fenghen 备注 WebAV 邀请进入微信群

Web 音视频系列文章

Plan

webav's People

Contributors

felixonmars avatar hughfenghen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

webav's Issues

两个mp4视频拼接时出现错误

经过简单测试,这是一个很棒的项目!速度超快!
但我在拼接某两个mp4文件时,发生错误,给作者反馈一下,以便能更好地完善这个优秀的项目。
以下是我的示例代码:
html
<video id="video" controls></video>

javascript

import {
    MP4Clip,
    ImgClip,
    createChromakey,
    Combinator,
    OffscreenSprite,
    fastConcatMP4,
} from '@webav/av-cliper';
const videoList = [
    'https://aisort.net/video/f1.mp4',
    'https://aisort.net/video/f2.mp4'
];
const width = 1920;
const height = 720;
async function start() {
    const coms = (await Promise.all(videoList.map(async (vurl) => (await fetch(vurl)).body)))
        .map((sbody) => {
            return new MP4Clip(sbody);
        })
        .map((clip, idx) => new OffscreenSprite(`spr${idx}`, clip))
        .map(async (spr, idx) => {
            const com = new Combinator({ width, height });
            await spr.ready;
            //spr.rect.x = idx * spr.rect.w;
            spr.rect.x = 0;
            //await com.add(imgSpr);
            await com.add(spr, { main: true });
            return com.output();
        });
    return await fastConcatMP4(coms);
}
start().then(async res => {
    const buffer = await new Response(res).arrayBuffer();
    const blob = new Blob([new Uint8Array(buffer)], {type: 'video/mp4'})
    const video = document.getElementById('video');
    video.src = URL.createObjectURL(blob);
});

错误截图:
image

这可能是由于视频编码导致的问题吗?如尝试使用demo中的 123.mp4、223.mp4、323.mp4 视频进行拼接,它是可以跑通的,但换了其它视频,就跑不通了。

这将是一个非常先进和有用的项目,但我如何将项目转换为js在浏览器中运行呢?

作者你好,感谢你的贡献,这是一个非常超前的项目,我简单地尝试过WebCodecs API的性能,它的性能不是wasm可比的,相信您的项目能解决许多问题。
我想提一个可能很愚蠢的问题,您的这个用ts编写的项目如何打包为js项目,以在浏览器中运行呢?(我从来没有接触过ts开发)。
例如我希望使用贵项目中的“fastConcatMP4”模块,我应该如何将它导出为js呢?

播放的速率如何控制

HI 我想问下av-cliper中decode-media.html的播放速率如何控制

// 最快速度渲染视频所有帧 while (true) { const { state, video: videoFrame } = await clip.tick(time) if (state === 'done') break if (videoFrame != null && state === 'success') { ctx.clearRect(0, 0, cvs.width, cvs.height) // 绘制到 Canvas ctx.drawImage(videoFrame, 0, 0, videoFrame.codedWidth, videoFrame.codedHeight) // 注意,用完立即 close videoFrame.close() } // 时间单位是 微秒,所以差不多每秒取 30 帧,丢掉多余的帧 time += 33000 } clip.destroy()
这里你提到了时间单位是 微秒,所以差不多每秒取 30 帧,丢掉多余的帧
如果我只是想按照正常速度播放这个视频 这里的time应该取多少 或者有什么公式来计算
如果我想要按照可变速率来播放视频,又要增加什么参数呢

编码器初始化报错

编码器初始化报错
image

因为 WebAV 写死了 codec(avc1.4D0032')
设备不支持默认编码 avc1.4D0032, 但支持 avc1.42E01F;
image

录制静态窗口(窗口画面长时间无变化)时,AVRecorder.outputStream 无数据输出

会议场景,使用electron方案录制窗口

const stream = await navigator.mediaDevices.getUserMedia({
			audio: false,
			video: {
				mandatory: {
					chromeMediaSource: 'desktop',
					chromeMediaSourceId: sourceId,
					minWidth: 1280,
					maxWidth: 1280,
					minHeight: 720,
					maxHeight: 720
				}
			}
		});

当窗口无内容变更时,通过outputStream读取数据时,无数据返回。

const recorder = new AVRecorder(stream, {
			width: this.resolution.width,
			height: this.resolution.height,
			expectFPS: 30
		});
const streamReader = recorder.outputStream!.getReader();
const readFrameStream = () => {
    streamReader?.read().then(({ done, value }) => {
      console.log('read record stream: ', done, value);
      if (done) {
         return;
      }
      readFrameStream();
     // .... save value to blob for write
    })

}

添加的素材只保留原比例缩放

大佬能不能将插入的素材,如:video,image和text只保留按原比例缩放。因为一不小心比例变了弄不回去只能删了再添加。

谢谢!

在 WebWorker 环境下 await clip.ready 一直报错

在worker 环境下使用,这一行代码会报错:
await clip.ready;
645 av-utils.ts:11 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'length') at av-utils.ts:11:25 at Array.map (<anonymous>) at xt (av-utils.ts:11:10) at mp4-clip.ts:104:26 at u (mp4-clip.ts:255:5) at h (mp4-clip.ts:249:12) at mp4-clip.ts:267:21

webm alpha信道保留支持

实际场景中存在基于webm的透明通道视频,视频中有人物形象以及透明背景,想基于该视频在其之上加上背景图,生成一个完整的带背景的视频,目前看WebAv应该还暂时不支持这种格式的解复用和最后的封装,想问下有什么比较好的解决思路

添加的素材需要复制粘贴功能

除了删除和恢复以外。对于添加的素材,如:image,text,video等可否增加复制粘贴功能,从而提高工作效率。

谢谢!

Recorded MP4 file lacks duration information

Hi, @hughfenghen

I used your latest demo to record an mp4, but the recorded mp4 file lacks duration information, which makes it impossible to seek in some players.
I couldn't find any information in your code or in the mp4box.js API on how to write duration information. I'm new to audio and video and would appreciate your guidance. Thank you!

Here is the recorded mp4:

WebAv-export-1689128970586.mp4
屏幕截图 2023-07-12 103104

基于canvas来生成mp4能否有更快速的解决方案

目前手头有个项目需要模仿类似剪映,腾讯智影等在线视频编辑,实现一个简易版在线视频编辑项目,要求是“纯前端“。
仓库项目中类似av-canvas的功能,但是目前av-canvas最终导出mp4的时候,需要从头到尾播放一遍,这个有些慢了,作者能否有快速一些导出mp4的方案实现

可以支持录制某个非Canvas元素为mp4视频么

请教一下大佬,webav能解决下面的问题么
image

  1. 这是一个图片播放器,是一个div元素包含了图片,select等元素,需要录制成视频
  2. 但是WebCodecs API貌似支持录制canvas元素或者整个窗口
  3. 现在是先html2canvas,再用Whammy生成webm,但视频清晰度会比较低

There is a memory leak issue in the recording function.

This issue can be reproduced using the demo at https://hughfenghen.github.io/WebAV/demo/record-usermedia.html

During the recording process, the memory usage keeps increasing. Although the browser's garbage collector runs, it is not able to completely free up the memory.

By debugging the code and temporarily disabling certain lines, the memory leak issue is mostly resolved. The problematic code seems to be related to the following line:

mp4File.addSample(trackId, s.data, s) 

The specific code location is as follows:
屏幕截图 2023-07-18 183324
屏幕截图 2023-07-18 183409

MP4Clip能否导出完整的音频pcm数据

我们的场景:
下载到完整的mp4资源,用mp4clip解码出音视频,
video 切片播放就行,
音频获取到完整的pcm数据直接播放更加流畅简洁

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.