Better How to guide? Step by step readme or something. about panohead HOT 12 OPEN

@OverwriteDev I've almost got it working using the EG3D + Deep3DFaceRecon code but it's slightly off

I have actually tried using Deep3DFaceRecon to estimate the pose, however, the result is a lot worse than what you can get by using the given pose (see below). I wonder if the authors could kindly provide a hint of what they used to estimate face poses?

given pose: deep3d pose:

Short answer is we use different cropping, centering, and pose estimation script other than Deep3DFaceRecon. And since that was a company service, I won't be able to share it. In fact, I myself also don't have the access now. Now I'm trying to find alternatives to achieve similar results but cannot guarantee either... Will keep you guys posted.

from panohead.

We update the scripts and example data for obtaining camera poses and cropping the images for PTI. See https://github.com/SizheAn/PanoHead/blob/main/3DDFA_V2_cropping/cropping_guide.md

from panohead.

@OverwriteDev I've almost got it working using the EG3D + Deep3DFaceRecon code but it's slightly off

from panohead.

@SizheAn , do you know if the estimation ws done using a single image? Or this is a case where multi-image is used , calculated the camera data and just one image was used?

I'm trying to figure out how it woudl be possible to get such precise data using only one photo.

Only single image. Pretty accurate if you can detect facial landmarks in the image. Our method is a combination of company's service + 3DDFA_V2 (https://github.com/cleardusk/3DDFA_V2). You can check their examples.

from panohead.

@SizheAn Thank you so much for the updates, this is great!!!

from panohead.

@Lustgard Currently we're missing the camera pose estimation stage which means you can't use any random input image. What I did for testing was to use Stable Diffusion + ControlNet to generate new people in the exact same pose as the example dataset images, so I could keep the camera values from the existing dataset.json. Someone wrote up the instructions on my Reddit post about it: https://www.reddit.com/r/StableDiffusion/comments/14h0pf4/comment/jpf5qea/

If you want to generate the 3D PLY mesh as well as the video, you can check the required changes and dependencies on my forked repo:
https://github.com/hack-mans/PanoHead

from panohead.

@hack-mans That forked repo is very helpful. Thank you for that.

from panohead.

This is fantastic @SizheAn , this was massively helpful.
Not only did I manage to get it up and running with the directions you provided but I'm also now more familiar with 3DDFA_V2 which is a pretty awesome project :D

from panohead.

Thanks! I was able to solve the wrong NumPy version with the help of ChatGPT.
For others who might need help:
"conda install numpy=1.22.3"
Simple yes, but I'm no linux master yet, so baby steps atm.

from panohead.

I wonder if it's possible to infer a camera pose if we know the orientation of the face, using MediaPipe's face mesh pose estimation as an example.

https://developers.google.com/mediapipe/solutions/vision/face_landmarker/python

from panohead.

@OverwriteDev I've almost got it working using the EG3D + Deep3DFaceRecon code but it's slightly off

I have actually tried using Deep3DFaceRecon to estimate the pose, however, the result is a lot worse than what you can get by using the given pose (see below). I wonder if the authors could kindly provide a hint of what they used to estimate face poses?

given pose:

deep3d pose:

from panohead.

@SizheAn , do you know if the estimation ws done using a single image?
Or this is a case where multi-image is used , calculated the camera data and just one image was used?

I'm trying to figure out how it woudl be possible to get such precise data using only one photo.

from panohead.