doubiiu / tooncrafter Goto Github PK
View Code? Open in Web Editor NEW[SIGGRAPH Asia 2024, Journal Track] ToonCrafter: Generative Cartoon Interpolation
Home Page: https://doubiiu.github.io/projects/ToonCrafter/
License: Apache License 2.0
[SIGGRAPH Asia 2024, Journal Track] ToonCrafter: Generative Cartoon Interpolation
Home Page: https://doubiiu.github.io/projects/ToonCrafter/
License: Apache License 2.0
@Doubiiu Hello ToonCrafter Team,
We are a team called AIGODLIKE, a group is dedicated to the study of animation and cg technology, and we are excited to share to users that we have discovered your innovative animation technology, ToonCrafter.
We are currently working on implementing ToonCrafter into ComfyUI, a node-based powerful and modular stable diffusion GUI.
We believe that integrating ToonCrafter into ComfyUI will greatly enhance animation capabilities and provide significant benefits to animation industry.
Please feel free to reach out if you have any questions or if there are any specific guidelines or licences we should follow during the integration process. We look forward to your feedback and are excited about the possibilities of this collaboration will bring.
Thank you for your time and consideration.
see it ToonCrafter
AE working on z of shape (1, 4, 32, 32) = 4096 dimensions.
(main checkpoint is already downloaded into checkpoints/tooncrafter_512_interp_v1
)seems like it's stuck somewhere on HF downloads. lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1221, in hf_hub_download
kind of stuff.
How to implement Reference-based Sketch Colorization?
Trying to run this on Runpod as my 3090 just keeps getting Out of memory. However runpod requires this to have the --listen flag. I tried running gradio_app.py with --listen but I get the error
usage: gradio_app.py [-h]
gradio_app.py: error: unrecognized arguments: --listen
root@502fd222ada8:/workspace/ToonCrafter#
I also tried to figure out where to add it in the python script but I can't so raising it as an issue.
Is it possible to use it in the mode of generating only one fps. What would x2 do to fps anime?
strong!how to use this model
I'm not sure if this would work, but wouldn't the cleanup (reference-based sketch colorization) part of the AI be easy to isolate? It'd also be lighter on system resources probably.
Many human animators like to rough animate but cleanup is tedious. The clean-up examples also look the best out of them all.
Inputting PNG sequences would be nice too.
How to use API remote call, is there any documentation?
be launched?
Only this steps do not work on Windows:
conda create -n tooncrafter python=3.8.5
conda activate tooncrafter
pip install -r requirements.txt
Install Visual Studio Installer -> Individual Components -> MSVC v141 - VS 2017 C++ x64/x86 build tools (v14.16)
conda create -n tooncrafter python=3.8.5
conda activate tooncrafter
pip install -r requirements.txt
...
I am not sure of why the short brief instructions are for...
I have no idea how to solve this. this happened after trying to run python gradio_app.py
H:\AI\ToonCrafter> python gradio_app.py
Traceback (most recent call last):
File "H:\AI\ToonCrafter\gradio_app.py", line 4, in
from scripts.gradio.i2v_test_application import Image2Video
File "H:\AI\ToonCrafter\scripts\gradio\i2v_test_application.py", line 5, in
from scripts.evaluation.funcs import load_model_checkpoint, save_videos, batch_ddim_sampling, get_latent_z
File "H:\AI\ToonCrafter\scripts\evaluation\funcs.py", line 4, in
from decord import VideoReader, cpu
ModuleNotFoundError: No module named 'decord'
H:\AI\ToonCrafter>
Hope to add support for ComfyUI
Hi,
Thanks for your great work. I was analyzing the code, and was curious where the temporal layers in the current training setup are frozen (my understanding is that they are frozen in training to ensure the motion prior is not lost) ? I don't see any location where the temporal layers are being frozen.
Do let me know which line of code i am missing.
Thanks!
Adi
I am trying to test the tool but after running the installation process it does not work when I click on generate.
In the console an exception is raised on cuda initialization (Torch not compiled with CUDA enabled)
My card is a 3080 with 10GB.
Below I attach the full log after running gradio app
(tooncrafter) C:\Users\user\dev\ToonCrafter>python gradio_app.py
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.0.0+cu118 with CUDA 1108 (you have 2.0.0+cpu)
Python 3.8.10 (you have 3.8.5)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
AE working on z of shape (1, 4, 32, 32) = 4096 dimensions.
checkpoints/tooncrafter_512_interp_v1/model.ckpt
>>> model checkpoint loaded.
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Global seed set to 123
start: talking man 2024-06-07 23:46:38
Traceback (most recent call last):
File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\gradio\queueing.py", line 532, in process_events
response = await route_utils.call_process_api(
File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\gradio\route_utils.py", line 276, in call_process_api
output = await app.get_blocks().process_api(
File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\gradio\blocks.py", line 1923, in process_api
result = await self.call_function(
File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\gradio\blocks.py", line 1509, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\anyio\_backends\_asyncio.py", line 2177, in run_sync_in_worker_thread
return await future
File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\anyio\_backends\_asyncio.py", line 859, in run
result = context.run(func, *args)
File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\gradio\utils.py", line 832, in wrapper
response = f(*args, **kwargs)
File "C:\Users\user\dev\ToonCrafter\scripts\gradio\i2v_test_application.py", line 51, in get_image
model = model.cuda()
File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\lightning_fabric\utilities\device_dtype_mixin.py", line 69, in cuda
device = torch.device("cuda", torch.cuda.current_device())
File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\torch\cuda\__init__.py", line 674, in current_device
_lazy_init()
File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\torch\cuda\__init__.py", line 239, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
Hi, I am finetuning your model and training the spatial layer. Can you share your tensorboard file about the loss change?
Hi,
I'm looking into using your project and I noticed a Windows-specific fork ( https://github.com/sdbds/ToonCrafter-for-windows ) that alters the installation process by setting the PowerShell execution policy to Unrestricted. This raises some security red flags for me.
I’d like to ask:
Your insights would really help me and other potential users decide whether to use this fork or wait for an official update.
Thanks for your work on this project and looking forward to your thoughts!
Error occurred when executing DownloadAndLoadDynamiCrafterModel:
local variable 'dcmodel' referenced before assignment
File "D:\Program Files\ComfyUI-aki-v1.3\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\Program Files\ComfyUI-aki-v1.3\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\Program Files\ComfyUI-aki-v1.3\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\Program Files\ComfyUI-aki-v1.3\custom_nodes\ComfyUI-DynamiCrafterWrapper\nodes.py", line 146, in loadmodel
return (dcmodel,)
Hey, thanks for a very impressive work, since you've fine-tuned from DynamicCrafter, which also has 576x1024 resolution model, should be expect a higher-res version of ToonCrafter any time soon? Or you're not planning to release it?
Forgive my clumsiness, I guess it's something that's still developing. It's an amazing tool, but if it could generate slightly larger images it could be used beyond the demo.
I also see that the brightness/tone/contrast in the video changes slightly with respect to the original images.
hello just have this erros when i try to run "python gradio_app.py":
(env) Z:\ia\toonmaker\ToonCrafter>python gradio_app.py
Traceback (most recent call last):
File "Z:\ia\toonmaker\ToonCrafter\gradio_app.py", line 4, in
from scripts.gradio.i2v_test_application import Image2Video
File "Z:\ia\toonmaker\ToonCrafter\scripts\gradio\i2v_test_application.py", line 3, in
from omegaconf import OmegaConf
File "Z:\ia\toonmaker\ToonCrafter\env\lib\site-packages\omegaconf_init_.py", line 1, in
from .base import Container, DictKeyType, Node, SCMode
File "Z:\ia\toonmaker\ToonCrafter\env\lib\site-packages\omegaconf\base.py", line 9, in
from antlr4 import ParserRuleContext
ModuleNotFoundError: No module named 'antlr4'
i already fixed the issue of decord : ModuleNotFoundError: No module named 'decord' with pip install decord
but this time i can not do in and it is not in requierement.txt
(just folowed the instructions from this guy:
https://www.youtube.com/watch?v=hc5nF6rGa68&t=183s 1:46-3:08)
Hello, I am trying to install ToonCrafter using the install.ps1 file, how can I fix this?
I am getting these errors:
Collecting torch==2.0.1
Downloading https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-win_amd64.whl (2619.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 GB 1.9 MB/s eta 0:00:00
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\Users\Nebula PC\AppData\Local\Temp\pip-build-tracker-_6miaosx\4efdef60987dfb791b1ce7a28586a1374a8c3154277ce13ec300c376'
check models...
Directory: C:\Users\Nebula PC\Desktop\Tooncrafter\ToonCrafter-for-windows
Mode LastWriteTime Length Name
d----- 2/06/2024 5:34 PM checkpoints
Downloading tooncrafter_512_interp_v1 model...
huggingface-cli : The term 'huggingface-cli' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
again.
At C:\Users\Nebula PC\Desktop\Tooncrafter\ToonCrafter-for-windows\install.ps1:24 char:5
huggingface-cli download --resume-download Doubiiu/ToonCrafter -- ...
~~~~~~~~~~~~~~~
Have you developed a full version of ToonCrafter yet? If you have already developed full version, we are very interested in understanding more about its capabilities and features
i'm trying to run this on sagemaker studio lab but since it has a limited storage i can't download the checkpoint in the usual way and need to use a little trick. When i download the moldel in that way it create a new folder inside the checkpoint one. is there a way to change where the model should be found in the codes?
start: smiling 2024-06-09 20:45:35
Traceback (most recent call last):
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\routes.py", line 384, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\blocks.py", line 1032, in process_api
result = await self.call_function(
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\blocks.py", line 844, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "C:\SD U V5.1\ToonCrafter\ToonCrafter-fp16\scripts\gradio\i2v_test_application.py", line 94, in get_image
cond_images = model.embedder(img_tensor.unsqueeze(0)) ## blc
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "C:\SD U V5.1\ToonCrafter\ToonCrafter-fp16\lvdm\modules\encoders\condition.py", line 339, in forward
z = self.encode_with_vision_transformer(image)
File "C:\SD U V5.1\ToonCrafter\ToonCrafter-fp16\lvdm\modules\encoders\condition.py", line 346, in encode_with_vision_transformer
if self.model.visual.input_patchnorm:
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1709, in getattr
raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'VisionTransformer' object has no attribute 'input_patchnorm'
I tried Updating torch 2.3.1+cu118, torchvision 0.18.1+cu118, open-clip-torch 2.24.0 it...but the same issues is same...😭😭
Hi!
I was curious how many A100/H100 GPU hours training took on the entire training corpus?
Thanks!
Adi
I'm trying to see how you color black and white animations. How do you add the sketch reference?
it's a great job, thanks your release. i want to know is there train code or only inference code release??
I saw that you collected 270K cartoon video clips in this paper, will you open this dataset in the future?
Hi. I have a desktop that has 2x Tesla T4s, and it should be working because it has 32G VRAM in total, while other people reported to have a 27G VRAM usage when inferring. It should work but when inferring, only 1 gpu has being used, which caused a cuda out of memory error, what parts of code should i edit so that it can work on multiple GPUs? Thanks in advance
Could u please provide a gui exe windows version of ToonCrafter in release section?
Hi there,
What are the hardware requirements to run this on a local machine?
Due to the recent EU laws being passed, I'd like to know what the dataset is that it's trained on. The applications of this is limited in my region if I cannot point out the datasets of the tool.
Hey. First of all - great job on this!
I feel like this tool has so much potential, but, like Stable Diffusion - it feels like most of it is hidden upon release.
Any plans on making documentation/tools for community to train their own checkpoints, or maybe loras/etc?
I'd be really interested in that.
If not though - still thank you for this awesome tool and special thanks for making it opensourced!
after running ToonCrafter, it just generates forever and nothing happens for almost 40 minutes.
My GPU is RTX 4070Ti.
AE working on z of shape (1, 4, 32, 32) = 4096 dimensions.
checkpoints/tooncrafter_512_interp_v1/model.ckpt
>>> model checkpoint loaded.
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Global seed set to 789
start: an anime scene 2024-06-01 11:15:03
C:\ToonCrafter-for-windows\venv\lib\site-packages\torchvision\transforms\functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True).
warnings.warn(
I see that model's size is 10Gb. How much vram it needs on inference, 16gb is enough?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.