snap-research / hyperhuman Goto Github PK

Thanks for your work, I am wondering if the MSCOCO 2014 Human Validation in the paper will be released soon?

Thanks for sharing nice work.

I wanted to ask what "replicate layers" means. Does replicating include the model weights? Or, does it simply mean replicating same blocks (architecture), and each branch has their own set of weights?

It seems like Figure 2. is showing that each expert branches have their own set of weights for (first few and last few Downsample /Upsample blocks), and the intermediate blocks are shared between the experts. But it was not clear from the paper itself, and so I would like some explanation.

If my understanding is correct, then how are the features from different branches fused together? Is it simple addition of features before passing the "shared" intermediate layers?

Thank you.

Hi, thanks for your great work. Please allow me to ask a question that might not relate closely to the paper.

In the paper, you finetune the whole unet with three expert modules, which results in high computational cost. Is it possible to only finetune the extra expert modules(normal and depth) if we assume the rgb branch is already well-trained? Could you please provide some insights on this? Thank you in advance.

Hi, thanks for your great work.

I am confused about how the depth map is processed. If I understand correctly, the depth map has only one channel. Then how can it be encoded by the vae encoder? Could you please provide more details? Thank you.

Can we just use text as input to enforce the joint learning of image appearance, spatial relationship, and geometry in a unified network?

Thank you for your excellent work. I have a question, since openpose-style skeleton refers to a skeleton with hand joints, a neck joint that connects nose joint, left and right shoulder joints. This topology is more easy-to-understand, and contains more details about hands, and other body parts. I wonder why you do not use the openpose style.
A example of openpose style is in the following image:

Thank you for your excellent work. May I ask if the skeleton input to the UNet also goes through the VaeEncoder? Is the operation the same as with the image? Looking forward to your reply.

as you say, you use different conv to convert different controls to same input size 128*128 and elementsize add them all to an encoder of SDXL, so is it the same archetecture with ControlNet but only changes the base model and increase some control types? Why you name it refiner?

Will you opensource code?