FInetuning the whole unet about hyperhuman HOT 2 CLOSED

Hi, many thanks for your interest in our work!

1. As we haven't run such an experiment, I'm not sure if it's OK to only finetune the replicated network layers. Intuitively, I think we should at least finetune some of the shared backbone layers. Otherwise, the intermediate features can not be updated to simultaneously output aligned RGB, depth, and normal.

2. There are certain ways to decrease the computational cost for your information. For example, a) change into a smaller backbone to trade-off the performance and memory cost, e.g., Small SD and Tiny SD. b) Use the efficient parameter finetuning techniques like LoRA, or just finetune some certain layers in the shared backbone. c) common engineering tricks to reduce memory consumption as used in diffusers, e.g., gradient checkpointing, gradient accumulation w/ smaller batch size, deepspeed model parallelism, lower fp precision like fp16, efficient_xformers, etc.

Best

from hyperhuman.

Thanks a lot for your comments.

from hyperhuman.