Shape issues with controlnet_shuffle about animatediff-cli-prompt-travel HOT 8 CLOSED

Aha, you're right!

I thought I had tried the others on their own but I guess I've always left 'shuffle' on because it seemed to work.

When I run without 'shuffle' the others work fine, but as soon as I enable 'shuffle' together with another ControlNet it crashes.

Thanks for your help!

For now just ignoring 'shuffle' is a fine solution for me.

from animatediff-cli-prompt-travel.

from animatediff-cli-prompt-travel.

Thanks for the report.
Could you zip and paste data/controlnet_image/test?
And I'd like you to describe the command as well.

from animatediff-cli-prompt-travel.

Hi, thanks for your help.

Here's a zip with my data/controlnet_image/test and config: my_controlnet_image_test_and_config.zip

The command I'm running is:

animatediff generate -c config/prompts/prompt_travel_blobcube.json -W 512 -H 512 -L 15 -C 15

from animatediff-cli-prompt-travel.

Looking at it now, I think I understand why the results of the GIF above were so strange in the first frames. The GIF was with -L 128, so only the first 15-20 frames were getting ControlNet guidance.

Maybe the shuffle ControlNet wasn't working well for that model? Or the ControlNet guidance scale was too high? Or maybe, even though it's shape doesn't cause an error, there is an issue with the shuffle ControlNet as well?

from animatediff-cli-prompt-travel.

Thanks. I was able to reproduce it in my environment.
It seems that "controlnet_shuffle" is not working properly.

And the start of the frame starts from 0. I think that's why the first frame is wrong.

from animatediff-cli-prompt-travel.

Just to clarify, while I think there might also be an issue with 'controlnet_shuffle', the other ControlNets crash for me.

If I enable only 'controlnet_canny', for example, then video generation fails completely with the traceback up above.

RuntimeError: shape mismatch: value tensor of shape [2, 1280, 1, 8, 8] cannot be broadcast to indexing result of shape [2, 1280, 1, 1, 1].

Do you want me to open a separate issue for this?

from animatediff-cli-prompt-travel.

As for shuffle, it seems that the diffusers implementation does not support multi controlnet, so it seems to be sort of expected that it will fail, but it is strange that canny does. I just tried it, and in my environment, if only canny is enabled, it succeeds.

from animatediff-cli-prompt-travel.