How to build a new person? about audio2photoreal HOT 8 CLOSED

The answer is probably that you can't - unless you have access to a 3D full body capture stage.

In order to build a new drivable identity, you need a 3D capture of that person, track it, run 3D reconstruction, fit the rig and compute joint angles as well as unwrapped textures. You would then use these assets to learn a new body decoder as described in https://arxiv.org/abs/2105.10441.

from audio2photoreal.

Is it possible to use openpose for human key capture and NerF-based human reconstruction to build a new person.

from audio2photoreal.

You'd have to collect a set of training data to learn the motion specifics of that new representation.
If you want to use the pretrained models, you'd have to match the representation of our avatars exactly.

from audio2photoreal.

The answer is probably that you can't - unless you have access to a 3D full body capture stage.

In order to build a new drivable identity, you need a 3D capture of that person, track it, run 3D reconstruction, fit the rig and compute joint angles as well as unwrapped textures. You would then use these assets to learn a new body decoder as described in https://arxiv.org/abs/2105.10441.

Thankes for your reply. I mean is whether the 3D full body capture stage can be replaced by openpose and 3D human body reconstruction, and if so, whether the details of the .npy file in the dataset can be supplemented.

from audio2photoreal.

Ah, I see. No, this will not be possible - or, it will require a new research effort way beyond what this approach is able to do right now. If I understand you correctly, you describe another body decoder that is driven by openpose keypoints. We don't have that pipeline built in this work but of course you can build it yourself.
If you don't have a 3D capture stage, you can just render our avatars from different viewpoints and treat the result as ground truth to build your openpose + NeRF method on top.

from audio2photoreal.

Ok, I'll try. Thank you for this wonderful project.

from audio2photoreal.

Anyone can please give me a detailed explanation of body_pose.npy, face_expression.npy and missing_face_frames.npy in dataset folder.

from audio2photoreal.

The data format is described here: https://github.com/facebookresearch/audio2photoreal?tab=readme-ov-file#dataset

Could you clarify what part exactly is unclear to you? Thanks!

from audio2photoreal.