Provide a version that maps to SMPL or Unity Mecanim biped rig animations about audio2photoreal HOT 5 CLOSED

@alexanderrichard
Could you clarify a bit the intermediate representations of each model. I think the final visulization is done with the avatar renderer from the Driving-Signal Aware Full-Body Avatars paper, which I think isn't directly portable to any other renderer that relies on a combination of mesh, skeleton bones and blend shapes. Whether SMPL, or any other output, it would be helpful to know what are the intermediate representations from each network. If they are "proprietary" latent values of an avatar renderer, it will be way more challenging to add any cross platform functionality.

I know I could probably dig into the code and paper a bit more and get the answer, but you can potentially save me a few hours, with a direct answer.

and of course congratulations on the paper and results!

from audio2photoreal.

Sure. The body decoder consumes body pose (as joint angles) and face expression codes (following https://arxiv.org/abs/1808.00362) as input, and is conditioned on a view direction from which the avatar can then be rendered.

There is no compatibility with blendshape models if you want to render textured avatars. For that, you would need to solve a retargeting problem, which will not be perfect and deteriorate quality. With this in mind, there are two (lengthy, complicated) ways to make this approach run on other representations:

1. Re-rigging/Re-targeting. Let's forget about the textures for a moment. The core problem to generalize to other body representations is the different rig. You'd have to build a mapping from the joint angles of our rig to the representation (typically also in joint angles, but usually a different number of joints) of your target rig. Probably our representation is not a superset of your target representation, so this might be difficult.
2. Fit a SMPL model to "ground truth". You can of course use the provided renderer and render ground truth avatars from different view-points. You can then use the resulting multiview dataset to fit a SMPL model as if it was a "true" multiview capture.

None of this is trivial unless you already have a body tracking pipeline for (2) from other multiview/3D projects you worked on.
Sorry that there is no "quick and easy" solution to this.

from audio2photoreal.

Hi! We currently don't have plans to provide a mapping to SMPL but since both the joint angles and the body renderer for this project are released in this repository, please feel free to contribute a mapping to this repo :)

from audio2photoreal.

there are many existing body tracking pipelines

is there a trivial solution for 1 (?)

from audio2photoreal.

I tried it, but it didn't work very well.
Code:
Dump motion: https://github.com/see2023/audio2photoreal
Send to MetaHuman: https://github.com/see2023/Bert-VITS2-ext
Demo: https://www.bilibili.com/video/BV1dg4y1m7sX/

from audio2photoreal.