AnimateDiff integration for ComfyUI, adapts from sd-webui-animatediff. Please read the original repo README for more information.
- Clone this repo into
custom_nodes
folder. - Download motion modules and put them under
comfyui-animatediff/models/
.
- Original modules: Google Drive | HuggingFace | CivitAI | Baidu NetDisk
- Community modules: manshoety/AD_Stabilized_Motion | CiaraRowles/TemporalDiff
- AnimateDiff v2 mm_sd_v15_v2.ckpt
Download motion LoRAs and put them under comfyui-animatediff/loras/
folder.
Note: LoRAs only work with AnimateDiff v2 mm_sd_v15_v2.ckpt module.
![image](https://private-user-images.githubusercontent.com/133728487/270393962-7a9f62f7-702e-48a4-934c-bbfe1e23aff2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NjIxNDgsIm5iZiI6MTcyNDQ2MTg0OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjcwMzkzOTYyLTdhOWY2MmY3LTcwMmUtNDhhNC05MzRjLWJiZmUxZTIzYWZmMi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQwMTEwNDhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04MmFkM2FiY2Y1OWU3NmU2ZDBmNzg5OTgzNmQ4OTNjMzVkNzZlZTdkOTE4OWM1YzJhMzIyYWFlYzA4OTY2N2Q4JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.XjUEB7A-R2TehzLY2-2ivvdNQwjzeZTp1gvsaW2X85o)
Workflow: lora.json
Samples:
|
|
|
|
The sliding window feature enables you to generate GIFs without a frame length limit. It divides frames into smaller batches with a slight overlap. This feature is activated automatically when generating more than 16 frames. To modify the trigger number and other settings, utilize the SlidingWindowOptions
node. See the sample workflow bellow.
![image](https://private-user-images.githubusercontent.com/133728487/268482819-9d756d01-ea45-4d1c-8e48-56f2725c7ca1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NjIxNDgsIm5iZiI6MTcyNDQ2MTg0OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDgyODE5LTlkNzU2ZDAxLWVhNDUtNGQxYy04ZTQ4LTU2ZjI3MjVjN2NhMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQwMTEwNDhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00MDZiODRhMmQyM2NhZWEyYTFmZmYwMDFiYjNkYTUwNTU0ZGE0OWM3NWE4MTUyZDZkNDYyNWI2ZjNlOGVlNDMxJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.KXTLCgsFr3n8Oi83b-8-KtFKiT1GXQF4uaTKjRPIsXw)
- Mostly the same with
KSampler
motion_module
: useAnimateDiffLoader
to load the motion moduleinject_method
: should left defaultframe_number
: animation lengthlatent_image
: You can pass anEmptyLatentImage
sliding_window_opts
: custom sliding window options
![image](https://private-user-images.githubusercontent.com/133728487/269430269-a352195d-f40c-494d-bd3d-30ee88174b88.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NjIxNDgsIm5iZiI6MTcyNDQ2MTg0OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMwMjY5LWEzNTIxOTVkLWY0MGMtNDk0ZC1iZDNkLTMwZWU4ODE3NGI4OC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQwMTEwNDhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kYzJiMjkzMDY4YzdkOTAwMjU0OGE4YjMwZTEyMGE1ZjFhMzk5YTljNDc5M2ViNjFkYjgzZjliMTY4YWU0ZWRkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.HW8nf1kYAisgNX5K_zXejYqe-keRRsp-smaHoyaUa-E)
- Combine GIF frames and produce the GIF image
frame_rate
: number of frame per secondloop_count
: use 0 for infinite loopsave_image
: should GIF be saved to diskformat
: supportsimage/gif
,image/webp
(better compression),video/webm
,video/h264-mp4
,video/h265-mp4
. To use video formats, you'll need ffmpeg installed and available inPATH
![image](https://private-user-images.githubusercontent.com/133728487/268482889-381c5acc-06ef-43da-ada0-3dc76f37a3e4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NjIxNDgsIm5iZiI6MTcyNDQ2MTg0OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDgyODg5LTM4MWM1YWNjLTA2ZWYtNDNkYS1hZGEwLTNkYzc2ZjM3YTNlNC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQwMTEwNDhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kZGRkM2U2Y2ExZGMxY2VlZjQ2NTVmNTMzMjc3ZTdiMDk5YmQwZTU5NjE5NDZjZWI1NzllOTA5YzZjOTdhY2UyJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.yV_9-FiejwZ3k9kwmkV31aGrkVmv5axdlfi7wRcg9_8)
Custom sliding window options
context_length
: number of frame per window. Use 16 to get the best results. Reduce it if you have low VRAM.context_stride
:- 1: sampling every frame
- 2: sampling every frame then every second frame
- 3: sampling every frame then every second frame then every third frames
- ...
context_overlap
: overlap frames between each window sliceclosed_loop
: make the GIF a closed loop, will add more sampling step
![image](https://private-user-images.githubusercontent.com/133728487/269431763-6679a8dd-bf96-419f-8934-ea2b046dd23c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NjIxNDgsIm5iZiI6MTcyNDQ2MTg0OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMxNzYzLTY2NzlhOGRkLWJmOTYtNDE5Zi04OTM0LWVhMmIwNDZkZDIzYy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQwMTEwNDhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04OTk5OTE1NmFhZmIwNjg5OTI2YjNlZGYwYWNlNGNkZDIzOTc0NGI0NmY4NzViYWJhOTc4M2UyZWIxYzk5MTRkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.UB5rvIwQcpvzphZ9DzyXhILWQdiByruSOY6JS4E0mfk)
Load GIF or video as images. Usefull to load a GIF as ControlNet input.
frame_start
: Skip some begining frames and start atframe_start
frame_limit
: Only takeframe_limit
frames
![image](https://private-user-images.githubusercontent.com/133728487/269429598-684176d5-6369-4a27-9f33-e721e0fe1876.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NjIxNDgsIm5iZiI6MTcyNDQ2MTg0OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDI5NTk4LTY4NDE3NmQ1LTYzNjktNGEyNy05ZjMzLWU3MjFlMGZlMTg3Ni5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQwMTEwNDhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hOWMxNDVkNTM2ZTYzZTBiNmJkNmViZGEzODFmNjkwMzQ1Y2Q5NDJkY2FkYWNhN2JiZWI3Y2JhZDM1ZDc0NWE5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.2puzvy6ZNhfsby6FCuEIz43zIOVaejA2SHlQWd9MErI)
![image](https://private-user-images.githubusercontent.com/133728487/268311503-b7164539-bc58-4ef9-b178-d914e833805e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NjIxNDgsIm5iZiI6MTcyNDQ2MTg0OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4MzExNTAzLWI3MTY0NTM5LWJjNTgtNGVmOS1iMTc4LWQ5MTRlODMzODA1ZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQwMTEwNDhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hNGY2NGU1ZTc2ZGVhYmY5YTQyZDY0MDY1NDZkN2FjYzdkNDhkMjgyMzZmYzUzNjFhMjhlZDdhNTM0MjkyODAwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.HnUrAMPdXEEpD33hztk-j3quJrfRpdT7ZLgHAEIUrt4)
Workflow: simple.json
Samples:
![image](https://private-user-images.githubusercontent.com/133728487/269432912-0f8bfb87-83cb-4119-9777-e3948ec0cb5c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NjIxNDgsIm5iZiI6MTcyNDQ2MTg0OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMyOTEyLTBmOGJmYjg3LTgzY2ItNDExOS05Nzc3LWUzOTQ4ZWMwY2I1Yy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQwMTEwNDhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kODVhNTJkOTNmYmNkNzQ2YjlhMWVlZjIyN2NjMTY1ZGU1NDY1NWVkZjZjZmViMDdkZmVhZWE4YjZjYmIzOTU0JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.rlw8UW8_t1sNT_vyCeypKI_kcsC0bjlLaStcuX14uKQ)
Workflow: sliding-window.json
Samples:
|
|
Upscale latent output using LatentUpscale
then do a 2nd pass with AnimateDiffSampler
.
![image](https://private-user-images.githubusercontent.com/133728487/268485612-987a1c5a-c1f8-4b24-8c62-f14496261d6c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NjIxNDgsIm5iZiI6MTcyNDQ2MTg0OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDg1NjEyLTk4N2ExYzVhLWMxZjgtNGIyNC04YzYyLWYxNDQ5NjI2MWQ2Yy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQwMTEwNDhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iZjVhMjkxMmUyMmM0ODQ1ZDkxZDBkYmI0Y2ZmZDg4ODNkNjI5YzU3NTVkYzFiYjBkODZlZDBjYTQxMzZiN2RmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.9fQTf8KK6GpItNLqli9E3Bh_-o-NiOSdCtCZ_-l3kR4)
Workflow: latent-upscale.json
You will need following additional nodes:
- Kosinkadink/ComfyUI-Advanced-ControlNet: Apply different weight for each latent in batch
- Fannovel16/comfyui_controlnet_aux: ControlNet preprocessors
- Use
LatentKeyframe
andTimestampKeyframe
from ComfyUI-Advanced-ControlNet to apply diffrent weights for each latent index. - Use 2 controlnet modules for two images with weights reverted.
Workflow: cn-2images.json
Samples:
|
|
|
|
Using a GIF (or video, or a list of images) as ControlNet input.
Workflow: cn-vid2vid.json
Samples:
|
|
It's an xformers
bug accidentally triggered by the way the original AnimateDiff CrossAttention is passed in. The current workaround is to disable xformers with --disable-xformers
when booting ComfyUI.
Work around:
- Shorter your prompt and negative prompt
- Reduce resolution. AnimateDiff is trained on 512x512 images so it works best with 512x512 output.
- Disable xformers with
--disable-xformers
See: continue-revolution/sd-webui-animatediff#31
Training data used by the authors of the AnimateDiff paper contained Shutterstock watermarks. Since mm_sd_v15 was finetuned on finer, less drastic movement, the motion module attempts to replicate the transparency of that watermark and does not get blurred away like mm_sd_v14. Try other community finetuned modules.