Code Monkey home page Code Monkey logo

Comments (7)

mrousavy avatar mrousavy commented on June 19, 2024

Why are you trying to send the Frame to JS?

from react-native-vision-camera.

hjawwad avatar hjawwad commented on June 19, 2024

I've to pass these frames to agora webRTC for video chat.

from react-native-vision-camera.

mrousavy avatar mrousavy commented on June 19, 2024

Yea well 4k RGB Frames are 33MB, so at 60 FPS that's 2 GB per second.

If you use a lower resolution (e.g. 1280x720) that's 3.6MB per Frame, at 30 FPS that's 110 MB per second.

I think you'd need to encode or compress them before sending them over the network.

Either way, the problem here is that ArrayBuffers cannot be shared between Worklet runtimes.

What you need to do is to send the Frame itself to JS first:

frame.incrementRefCount() // <-- otherwise Frame is deleted by the time `something` is called
runOnJS(something)(frame)

and then in something you do:

const array = frame.toArrayBuffer()
frame.decremetRefCount() // <-- otherwise camera stalls

from react-native-vision-camera.

hjawwad avatar hjawwad commented on June 19, 2024

I tried it out but it's producing the errors.

  const receiveFrame = (frame: any) => {
      if (frame.pixelFormat === 'rgb') {
        const buffer = frame.toArrayBuffer();
        const data = new Uint8Array(buffer);
        frame.decremetRefCount();
        console.log(`Pixel at 0,0: RGB(${data[0]}, ${data[1]}, ${data[2]})`);
      }
    };
  
    const frameProcessor = useFrameProcessor((frame: any) => {
      'worklet';
      frame.incrementRefCount();
      runOnJS(receiveFrame)(frame);
    }, []);

I change receiveFrame as worklet but it dosn't work.

Screenshot 2024-05-14 at 7 10 57 PM

Here is the repo link to checkout complete code
https://github.com/hjawwad/VideoFrames/blob/master/App.tsx

from react-native-vision-camera.

mrousavy avatar mrousavy commented on June 19, 2024

runOnJS is from Reanimated. We are using react-native-worklets-core, so you need to do Worklets.createRunOnJS.

from react-native-vision-camera.

shaheer741 avatar shaheer741 commented on June 19, 2024

@hjawwad
were you able to resolve your issue?

from react-native-vision-camera.

hjawwad avatar hjawwad commented on June 19, 2024

Yes, but it's only processing 5 or 7 frames per second. On average it's taking 200ms per frame.
@mrousavy Is this expected behaviour or it's taking more time for me.

const pushVideoFrame = Worklets.createRunOnJS(
  async (resized: any, frame: FrameInternal) => {
    console.log(`Pixel in RGB ${resized[0]}, ${resized[1]}, ${resized[2]}`);
    frame.decrementRefCount();
  },
);

const frameProcessor = useFrameProcessor((frame: any) => {
  'worklet';
  runAtTargetFps(20, () => {
    'worklet';
    const start = performance.now();
    if (frame.pixelFormat === 'yuv') {
      const buffer = resize(frame, {
        scale: {
          width: 50,
          height: 100,
        },
        pixelFormat: 'rgba',
        dataType: 'uint8',
      });

      frame.incrementRefCount();
      pushVideoFrame(buffer!, frame);
      const end = performance.now();

      const diff = (end - start).toFixed(2);
      console.log(`Push Video Frames took ${diff}ms!`);
    }
  });
}, []);
Screenshot 2024-05-22 at 6 31 22 PM

from react-native-vision-camera.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.