Hello, I have two questions. Can you help me answer them （3D face tracking and headpose） about livespeechportraits HOT 9 OPEN

3D facial tracking is used to obtain 3D facial landmarks and head poses. Deca is an alternative option to get these parameters. In fact, any parametric 3D facial reconstruction method is considerable. Camera calibration is used to achieve better reconstruction.

from livespeechportraits.

Thanks，I still have some questions about the two preprocessed data. How can I get the id_scale.mat and shoulder_ Point3d (I know this is calculated through LK optical flow, but how to get the key points of 3D)

from livespeechportraits.

id & scale parameters are results of 3D facial tracking.
To get 3D shoulder points, we first detect 2D shoulder points using LK flow and reconstruct 3D shoulder points by assuming a billboard model where the depth is the mean 3D facial depth for each frame.

from livespeechportraits.

thank you again ！ I trained the audio2feature and audio2headpose modules. Here are some questions I hope you can answer

1. How can I get a linear velocity term in Δ Pt audio2headpose

2. Whether tracked 3D in audio2feature is normalized? I used [0,1] normalization. It seems that there are some problems

3. In the article, proj in Fig. 2 stands for camera projection. How can I integrate the mouth headpose and eye information through this? My understanding is to convert 3D into 2D through the camera. So the rest of the landmark information adopts the original landmark of the video？

4. Whether HeadPose is relevant with contour（landmark） ？

from livespeechportraits.

1. velocity head pose is just the delta speed, velocity_t = pose_t+1 - pose_t. It is an implicit supervision term. You can define it as you will.
2. The 3D facial landmarks lie in the linear 3DMM space, I didn't do additional normalization on it. The template face model is normalized in a fixed space actually for any 3DMM model.
3. Camera projection depends on the 3d face tracking algorithm you used. Landmarks like eyes can just be sampled from the dataset, of course in 3D space. All these should be done before the projection.
4. Head pose is the tracking results of the reconstruction results. No relationship exists between the pose and the contour landmarks.

from livespeechportraits.

1. As for 3D face tracking, whether it is obtained from the original image（original video is 1920*1080）and whether it is affected by camera calibration, If I using deca to calculate the 3D key points for each frame，what is the different with two methods?
2. What method is used to obtain GroundTruth of the HeadPose? （ I use OpenFace2.2.0 method to get the HeadPose GroundTruth ）

Thank you again for publishing the code. Thank you very much！

Hello, I have two questions, can you help me. I use the method of OpenFace2.2.0 to get the HeadPose GroundTruth.

1. How can I get the tracked3D_normalized_pts_fix_contour.npy and 3d_fit_data.npz
2. And I know what they mean?

thank you so much for your marvelous work.

from livespeechportraits.

These two files are 3d facial tracking results (3D landmarks, head poses, etc.). Check the inference code and replace them with your tracking results works.

from livespeechportraits.

As I understand, the papers 1 and 2 (mentioned in paragraph 2 of sec 4.1) are used to extract the face landmarks. Did you just use the paper codes, or there are off-the-shelf tools in python?

[1] Automatic Acquisition of High-fidelity Facial Performances Using Monocular Videos.
[2] Face2Face: Real-time Face Capture and Reenactment of RGB Videos.

from livespeechportraits.

As I understand, the papers 1 and 2 (mentioned in paragraph 2 of sec 4.1) are used to extract the face landmarks. Did you just use the paper codes, or there are off-the-shelf tools in python?

[1] Automatic Acquisition of High-fidelity Facial Performances Using Monocular Videos. [2] Face2Face: Real-time Face Capture and Reenactment of RGB Videos.

maybe you can see, foocker/LSP

from livespeechportraits.