Hello keijiro,
Hope you are fine, I am new to Unity and c-sharp so can you provide ideas as to how I can input a video clip instead of a webcam input. If you have done any work on it, that can be very helpful as I would have direct ideas of how to integrate it.

Hi keijiro,

Thank you for the amazing tool!
Today I try to grab the faceLandmarks data in the FacePipeline from c# script, and it works well.

And inside the _pipeline.RefinedFaceVertexBuffer, there are total 936 float4,
which the first 468 are faceLandmarks for the faceMesh, and there are another 468 float4 I couldn't figure out what they are.

They seems to have a face shape, but only have values when I move. Are they something like motion vectors?

Just curious, and thank you for the time!

Hey there - thanks for sharing this awesome example. I want to attach 3D models to the tracked face. Is it possible to figure out the center of the head position in world space using this method?

Visualizer scene looks like this:

Unity 2020.3.24f1
Target: iOS
Device: MacBook Pro 2019

The only error in console is

Metal: Error creating pipeline state (Hidden/MediaPipe/FaceMesh/Preprocess): Link failed: The type of fragment input user(TEXCOORD0) does not match the type of the corresponding vertex shader output
(null)

Would be glad to provide any additional information.

I am tryint to browse the code to understanding more for the purpose of implementing something similar to the mediapipe effect renderer. However, it's a bit challenging browsing it because for some reason, unity is not creating the .sln file. And when I try to open as a C# project, it gives the error below:

I am on windows 11, Unity 2021.1.3f1, Visual Studio 2022

I have searched for info about this error, and they usually say to try to fix the file locations of the files specified in the error. However, as you can see above, the error message is just blank, no files were specified. Do you have an idea on how to fix this or maybe a workaround?
thanks!

We have a custom mesh for our project, is there a way to load that custom face mesh to use? Its currently setup with a neck and head bone, and it has blendshapes for face movements.

Hi
often RefinedFaceMesh were broken
after rebooting the unity editor, it worked fine.
but I don't know the reason
Thanks for helping me

Hello,

We noticed the algorithm with face recognition works when the face is tilted not more than ~45 degrees. When the angle is more significant it stops working

Would it be possible whether the algorithm works with a face that is upside down?
If so what changes would be required?

Thanks in advance,
MD

Hi @keijiro ,

Hope you are all well !

I was wondering if it is possible to cumulate facemesh and openpose on a unity chan avatar from a video input.

Like, https://github.com/digital-standard/ThreeDPoseUnityBarracuda, and your repository ?

Thanks for any insights or inputs on that.

Ps. Your repositories are just awesome, thanks for sharing with us.

Cheers,
Luc Michalski

I've tried to change the input webcam resolution to 1920x1080, but the mesh position is incorrect and it is seems only crop the center part for the face detection processing. Also, the detection is not as smooth as the sample.

Hello, wanted to know if its possible to share fbx/obj model of the TemplateMesh.asset file. We need to modify the mesh slightly and cant seems to extract/convert to fbx/obj formats as the TemplateMesh.asset file is packaged.

By integrating the new face_blendshapes.tflite into your project, the generated 52 blendshapes coefficients could be used to drive the face of the 3D avatars from ReadyPlayer.ME.

Latest Documentation for April 2023

Hi,

very nice project, but I have some issues running it (with camera and based on the provided png files).
As you see from the pictures, the rendering of the mesh doesn't work as expected.
I have only downloaded the project and opened it in Unity, running the Visualizer scene.

OS: Windows 10
Unity version: 2021.1.3
Intel Iris Xe Grafik

Hi @keijiro, thank you for the great project.
I am currently using this project to do research on mouth movements.
My question is: I would like to save my mouth movements as a dataset using this project.
I would like to know if it is possible to do that.

I am waiting for your reply. Thank you in advance.

Hi,

thank you for this project :-)

I'm new to Unity ...

I had a problem ... the camera view is shown (Main object ) but only this one ... no other objects (Face, Left Eye & Right Eye) are rendered ... also the meshes are not rendered at all ... so only the camera view is visible ...

the method Visualizer::OnRenderObject is called and the calculations there looks fine (I debugged them) ... so not sure what is the problem ....

Im using 2021.1.3f1 ...

Could you please give some tips what should I check?

Thanks in advance,
Best Regards,
MD

Shader is failing on launch with error (Unity Editor)

Metal: Error creating pipeline state (Hidden/MediaPipe/FaceMesh/Preprocess): Link failed: The type of fragment input user(TEXCOORD0) does not match the type of the corresponding vertex shader output
(null)

Unity 2020.3.24f1
Target: iOS

Hi again,

More contextualzed question, is it possible to animate unity chan face with your repository ?

Cheers,
Luc Michalski

I am trying to export the project with the dependencies but it doesn't seem to export all the necessary dependcies.

When I import it into a different project, I get dependencies error. Can you please provide a solution?
Thank you

Thanks for your great work.
I want to make a face texture for the face mesh. I set the this.GetComponent<MeshFilter>().mesh = _resources.faceMeshTemplate; in Visualizer.cs OnPostRender() function. but it not work. By the way, I found the _resources.faceMeshTemplate.Vertices are all zero. So which mesh can set UVs and be shown?
I am not very familiar with unity and CG. The above statement may be not very accurate. Could you please provide some ideas for that task? Thanks!

example about what I want to implement

Hi @keijiro, thank you for the great project. I've run the project on Android and I'm getting much lower FPS vs Unity Editor. Could I ask for any advice to tune the performance for Android? This project uses computeshader and much of the advice for optimizing Barracuda is for creating and running tensors.

I've tested the Google mediapipe demo and the performance is closer to that of unity editor demo, so it seems like its possible to reach that level optimization on the same mobile hardware.

I thought _pipeline.RawLeftEyeVertexBuffer has 67 vertices.
but only 5 points have 3d locations.
Do i miss something?
thanks for your answer.

Thanks for sharing this project, these samples are super valuable. Amazing stuff!

I have managed to run FaceMeshBarracuda locally without issue. However, I noticed that the face mesh is essentially a 2D projection distorted/scaled to fit the webcam image. What I hoped to accomplish was to estimate the actual world-space position/rotation of the face (even if the unit scale wouldn't be accurate). I don't actually need the mesh, just tracking of the approximate head transform.

I am assuming that won't be possible to estimate using this package. Given that you've been porting a lot of similar packages to Barracuda recently, do you know of any other Barracuda-compatible models I could experiment with?

Thanks for the awesome project. I was interested in trying to translate this to work with facial blend shapes on a 3D characters. Any examples or ideas for how to do that?