Comments (7)
Why are you trying to send the Frame to JS?
from react-native-vision-camera.
I've to pass these frames to agora webRTC for video chat.
from react-native-vision-camera.
Yea well 4k RGB Frames are 33MB, so at 60 FPS that's 2 GB per second.
If you use a lower resolution (e.g. 1280x720) that's 3.6MB per Frame, at 30 FPS that's 110 MB per second.
I think you'd need to encode or compress them before sending them over the network.
Either way, the problem here is that ArrayBuffer
s cannot be shared between Worklet runtimes.
What you need to do is to send the Frame itself to JS first:
frame.incrementRefCount() // <-- otherwise Frame is deleted by the time `something` is called
runOnJS(something)(frame)
and then in something
you do:
const array = frame.toArrayBuffer()
frame.decremetRefCount() // <-- otherwise camera stalls
from react-native-vision-camera.
I tried it out but it's producing the errors.
const receiveFrame = (frame: any) => {
if (frame.pixelFormat === 'rgb') {
const buffer = frame.toArrayBuffer();
const data = new Uint8Array(buffer);
frame.decremetRefCount();
console.log(`Pixel at 0,0: RGB(${data[0]}, ${data[1]}, ${data[2]})`);
}
};
const frameProcessor = useFrameProcessor((frame: any) => {
'worklet';
frame.incrementRefCount();
runOnJS(receiveFrame)(frame);
}, []);
I change receiveFrame
as worklet but it dosn't work.
![Screenshot 2024-05-14 at 7 10 57 PM](https://private-user-images.githubusercontent.com/61308073/330490814-761c42a8-7fdd-4ddc-9f00-109cfbc36121.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTY5MjI5MzYsIm5iZiI6MTcxNjkyMjYzNiwicGF0aCI6Ii82MTMwODA3My8zMzA0OTA4MTQtNzYxYzQyYTgtN2ZkZC00ZGRjLTlmMDAtMTA5Y2ZiYzM2MTIxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTI4VDE4NTcxNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY0YTQ5NzIzNmQ1MTQwZjQ4MjA3OTM1YzJlZjY3ZDQ3YWZmN2MwOTliOWFmMjk4ODUxZjRmNjA0YWUxYWYxNTgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.BwVKVhmNUctYzbkUJs6ZGY6MZwbgAAH0IiwYUkZXK1U)
Here is the repo link to checkout complete code
https://github.com/hjawwad/VideoFrames/blob/master/App.tsx
from react-native-vision-camera.
runOnJS
is from Reanimated. We are using react-native-worklets-core, so you need to do Worklets.createRunOnJS
.
from react-native-vision-camera.
@hjawwad
were you able to resolve your issue?
from react-native-vision-camera.
Yes, but it's only processing 5 or 7 frames per second. On average it's taking 200ms per frame.
@mrousavy Is this expected behaviour or it's taking more time for me.
const pushVideoFrame = Worklets.createRunOnJS(
async (resized: any, frame: FrameInternal) => {
console.log(`Pixel in RGB ${resized[0]}, ${resized[1]}, ${resized[2]}`);
frame.decrementRefCount();
},
);
const frameProcessor = useFrameProcessor((frame: any) => {
'worklet';
runAtTargetFps(20, () => {
'worklet';
const start = performance.now();
if (frame.pixelFormat === 'yuv') {
const buffer = resize(frame, {
scale: {
width: 50,
height: 100,
},
pixelFormat: 'rgba',
dataType: 'uint8',
});
frame.incrementRefCount();
pushVideoFrame(buffer!, frame);
const end = performance.now();
const diff = (end - start).toFixed(2);
console.log(`Push Video Frames took ${diff}ms!`);
}
});
}, []);
![Screenshot 2024-05-22 at 6 31 22 PM](https://private-user-images.githubusercontent.com/61308073/332816404-7242b64e-b8a4-47a2-991f-4b1512565d8e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTY5MjI5MzYsIm5iZiI6MTcxNjkyMjYzNiwicGF0aCI6Ii82MTMwODA3My8zMzI4MTY0MDQtNzI0MmI2NGUtYjhhNC00N2EyLTk5MWYtNGIxNTEyNTY1ZDhlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTI4VDE4NTcxNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEwMzQwMDg5OTExNzY0NTEzODQyMGJmOWFjMWQ4MmVkMTViZTUzMzNjNjgwMzhlNjI5YjEwYjYyZWI4OWZkN2MmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0._mqQwHiSjPWJBAvRCZjBr17Yj-MoDwopn3fDZUfhIbI)
from react-native-vision-camera.
Related Issues (20)
- 💭 Two separate frame processors at different target FPS HOT 7
- 🐛 [AVCaptureDevice setActiveColorSpace:] Not supported - use activeFormat.supportedColorSpaces'. HOT 2
- 🐛 APK Size increase After implementing react-native-vision-camera lib double the apk size HOT 5
- 💭 Why the Photo Result Does Not Come Out in the Specified Aspect Ratio HOT 1
- 🐛 `finally` on a `Promise` stops camera on Android HOT 3
- 🐛 [V4][Android] Release crash: maxImages (6) has already been acquired HOT 1
- 🔧 Not able to build expo dev client when using any of the given frame processing plugin for OCR HOT 2
- 🐛 [Android] isActive change doesn't happen fast enough causing device/camera-already-in-use error when switching to native camera app HOT 1
- 🔧 Execution failed for task ':react-native-vision-camera:compileReleaseKotlin'. HOT 3
- 🔧 This API is provided only for React Native frameworks and not intended for general users. This API can change between minor versions in alignment with React Native frameworks and won't be considered a breaking change. HOT 10
- 💭 Android: Frameprocessor HOT 7
- 💭 Focus not work HOT 1
- 🐛 Memory leak Android HOT 6
- 💭 [iOS] Did someone manage to get to reasonable FPS with ML Kit Pose Detection (native frame processor plugin)? HOT 2
- 💭 Lifecycle methods for native frame processors HOT 5
- 💭 How to access GPS location tags in photo metadata? HOT 4
- 🐛 takePhoto() function working very slowly on Android HOT 3
- 🔧 A failure occurred while executing org.jetbrains.kotlin.compilerRunner.GradleCompilerRunnerWithWorkers$GradleKotlinCompilerWorkAction HOT 12
- 💭 How does RNVC interact with OS camera settings? HOT 1
- 💭 Is there a migration guide from V3 to V4? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from react-native-vision-camera.