keijiro / facemeshbarracuda Goto Github PK
View Code? Open in Web Editor NEWLightweight facial capture package for Unity
License: Apache License 2.0
Lightweight facial capture package for Unity
License: Apache License 2.0
Thanks for sharing this project, these samples are super valuable. Amazing stuff!
I have managed to run FaceMeshBarracuda locally without issue. However, I noticed that the face mesh is essentially a 2D projection distorted/scaled to fit the webcam image. What I hoped to accomplish was to estimate the actual world-space position/rotation of the face (even if the unit scale wouldn't be accurate). I don't actually need the mesh, just tracking of the approximate head transform.
I am assuming that won't be possible to estimate using this package. Given that you've been porting a lot of similar packages to Barracuda recently, do you know of any other Barracuda-compatible models I could experiment with?
Hi @keijiro, thank you for the great project.
I am currently using this project to do research on mouth movements.
My question is: I would like to save my mouth movements as a dataset using this project.
I would like to know if it is possible to do that.
I am waiting for your reply. Thank you in advance.
Shader is failing on launch with error (Unity Editor)
Metal: Error creating pipeline state (Hidden/MediaPipe/FaceMesh/Preprocess): Link failed: The type of fragment input user(TEXCOORD0) does not match the type of the corresponding vertex shader output
(null)
Unity 2020.3.24f1
Target: iOS
Visualizer scene looks like this:
Unity 2020.3.24f1
Target: iOS
Device: MacBook Pro 2019
The only error in console is
Metal: Error creating pipeline state (Hidden/MediaPipe/FaceMesh/Preprocess): Link failed: The type of fragment input user(TEXCOORD0) does not match the type of the corresponding vertex shader output
(null)
Would be glad to provide any additional information.
I am tryint to browse the code to understanding more for the purpose of implementing something similar to the mediapipe effect renderer. However, it's a bit challenging browsing it because for some reason, unity is not creating the .sln file. And when I try to open as a C# project, it gives the error below:
I am on windows 11, Unity 2021.1.3f1, Visual Studio 2022
I have searched for info about this error, and they usually say to try to fix the file locations of the files specified in the error. However, as you can see above, the error message is just blank, no files were specified. Do you have an idea on how to fix this or maybe a workaround?
thanks!
Hi keijiro,
Thank you for the amazing tool!
Today I try to grab the faceLandmarks data in the FacePipeline from c# script, and it works well.
And inside the _pipeline.RefinedFaceVertexBuffer, there are total 936 float4,
which the first 468 are faceLandmarks for the faceMesh, and there are another 468 float4 I couldn't figure out what they are.
They seems to have a face shape, but only have values when I move. Are they something like motion vectors?
Just curious, and thank you for the time!
We have a custom mesh for our project, is there a way to load that custom face mesh to use? Its currently setup with a neck and head bone, and it has blendshapes for face movements.
Hello keijiro,
Hope you are fine, I am new to Unity and c-sharp so can you provide ideas as to how I can input a video clip instead of a webcam input. If you have done any work on it, that can be very helpful as I would have direct ideas of how to integrate it.
I am trying to export the project with the dependencies but it doesn't seem to export all the necessary dependcies.
When I import it into a different project, I get dependencies error. Can you please provide a solution?
Thank you
Hi @keijiro ,
Hope you are all well !
I was wondering if it is possible to cumulate facemesh and openpose on a unity chan avatar from a video input.
Like, https://github.com/digital-standard/ThreeDPoseUnityBarracuda, and your repository ?
Thanks for any insights or inputs on that.
Ps. Your repositories are just awesome, thanks for sharing with us.
Cheers,
Luc Michalski
By integrating the new face_blendshapes.tflite into your project, the generated 52 blendshapes coefficients could be used to drive the face of the 3D avatars from ReadyPlayer.ME.
Hi,
thank you for this project :-)
I'm new to Unity ...
I had a problem ... the camera view is shown (Main object ) but only this one ... no other objects (Face, Left Eye & Right Eye) are rendered ... also the meshes are not rendered at all ... so only the camera view is visible ...
the method Visualizer::OnRenderObject is called and the calculations there looks fine (I debugged them) ... so not sure what is the problem ....
Im using 2021.1.3f1 ...
Could you please give some tips what should I check?
Thanks in advance,
Best Regards,
MD
Thanks for your great work.
I want to make a face texture for the face mesh. I set the this.GetComponent<MeshFilter>().mesh = _resources.faceMeshTemplate;
in Visualizer.cs OnPostRender()
function. but it not work. By the way, I found the _resources.faceMeshTemplate.Vertices
are all zero. So which mesh can set UVs and be shown?
I am not very familiar with unity and CG. The above statement may be not very accurate. Could you please provide some ideas for that task? Thanks!
Hello, wanted to know if its possible to share fbx/obj model of the TemplateMesh.asset file. We need to modify the mesh slightly and cant seems to extract/convert to fbx/obj formats as the TemplateMesh.asset file is packaged.
Hello,
We noticed the algorithm with face recognition works when the face is tilted not more than ~45 degrees. When the angle is more significant it stops working
Would it be possible whether the algorithm works with a face that is upside down?
If so what changes would be required?
Thanks in advance,
MD
Thanks for the awesome project. I was interested in trying to translate this to work with facial blend shapes on a 3D characters. Any examples or ideas for how to do that?
Hi again,
More contextualzed question, is it possible to animate unity chan face with your repository ?
Cheers,
Luc Michalski
Hi,
very nice project, but I have some issues running it (with camera and based on the provided png files).
As you see from the pictures, the rendering of the mesh doesn't work as expected.
I have only downloaded the project and opened it in Unity, running the Visualizer scene.
OS: Windows 10
Unity version: 2021.1.3
Intel Iris Xe Grafik
I've tried to change the input webcam resolution to 1920x1080, but the mesh position is incorrect and it is seems only crop the center part for the face detection processing. Also, the detection is not as smooth as the sample.
Hi Keijiro,
First of all, thank you for this amazing project! I have been experimenting with this repo, and it works so well. However, I am interested in using this to predict the direction of a player's iris ( left, right, up, down).
I try to get the data from the graphic buffer _pipeline.RawLeftEyeVertexBuffer
, but i have 0.5 knowledge on compute shader
Could you please provide some guidance or example code on how to achieve this?
Thank you for your time and assistance!
paopeaw
Hi @keijiro, thank you for the great project. I've run the project on Android and I'm getting much lower FPS vs Unity Editor. Could I ask for any advice to tune the performance for Android? This project uses computeshader and much of the advice for optimizing Barracuda is for creating and running tensors.
I've tested the Google mediapipe demo and the performance is closer to that of unity editor demo, so it seems like its possible to reach that level optimization on the same mobile hardware.
Hey there - thanks for sharing this awesome example. I want to attach 3D models to the tracked face. Is it possible to figure out the center of the head position in world space using this method?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.