The tooncrafter_with_sketchguidance from mattyamonaca

tooncrafter_with_sketchguidance's Introduction

ToonCrafter_with_SketchGuidance

This repository is an implementation that recreates the SketchGuidance feature of "ToonCrafter".

2024-07-13.2.52.44.mov

🧰 Models

Model	Resolution	GPU Mem. & Inference Time (A100, ddim 50steps)	Checkpoint
ToonCrafter_512	320x512	TBD (`perframe_ae=True`)	Hugging Face
SketchEncoder	TBD	TBD	Hugging Face

Currently, ToonCrafter can support generating videos of up to 16 frames with a resolution of 512x320. The inference time can be reduced by using fewer DDIM steps.

⚙️ Setup

Install Environment via Anaconda (Recommended)

conda create -n tooncrafter python=3.8.5
conda activate tooncrafter
pip install -r requirements.txt

💫 Inference

1. Local Gradio demo

Download pretrained ToonCrafter_512 and put the model.ckpt in checkpoints/tooncrafter_512_interp_v1/model.ckpt.
Download pretrained SketchEncoder and put the model.ckpt in control_models/sketch_encoder.ckpt.

  python gradio_app.py

tooncrafter_with_sketchguidance's People

Contributors

Stargazers

Watchers

tooncrafter_with_sketchguidance's Issues

can you also do it with colored input image only?

(waiting for Kijai comfyui node update)I can't test it out yet, since I have low VRAM, but i remember seeing a demo showing that you only need 1 fully colored frame, does that also work or do you need both the starting and ending frame?

Recommend Projects