โฌDownload Models | ๐ปHow to Test | ๐ฅ
Official implementation of T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models.
We propose T2I-Adapter, a simple and small (~70M parameters, ~300M storage space) network that can provide extra guidance to pre-trained text-to-image models while freezing the original large text-to-image models.
T2I-Adapter aligns internal knowledge in T2I models with external control signals. We can train various adapters according to different conditions, and achieve rich control and editing effects.
Put the downloaded models in the T2I-Adapter/models
folder.
- The T2I-Adapters can be download from https://huggingface.co/TencentARC/T2I-Adapter.
- The pretrained Stable Diffusion v1.4 models can be download from https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/tree/main. You need to download the
sd-v1-4.ckpt
file. - [Optional] If you want to use Anything v4.0 models, you can download the pretrained models from https://huggingface.co/andite/anything-v4.0/tree/main. You need to download the
anything-v4.0-pruned.ckpt
file. - The pretrained clip-vit-large-patch14 folder can be download from https://huggingface.co/openai/clip-vit-large-patch14/tree/main. Remember to download the whole folder!
- The pretrained keypose detection models include FasterRCNN (human detection) from https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth and HRNet (pose detection) from https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth.
After downloading, the folder structure should be like this:
- Python >= 3.6 (Recommend to use Anaconda or Miniconda)
- PyTorch >= 1.4
pip install -r requirements.txt
- If you want to use the full function of keypose-guided generation, you need to install MMPose. For details please refer to https://github.com/open-mmlab/mmpose.
- The results are in the
experiments
folder. - If you want to use the
Anything v4.0
, please add--ckpt models/anything-v4.0-pruned.ckpt
in the following commands.
python app.py
- Sketch to Image Generation
python test_sketch.py --plms --auto_resume --prompt "A car with flying wings" --path_cond examples/sketch/car.png --ckpt models/sd-v1-4.ckpt --type_in sketch
- Image to Image Generation
python test_sketch.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/anything_sketch/human.png --ckpt models/sd-v1-4.ckpt --type_in image
- Generation with Anything setting
You can use gradio to experience all these three functions at once. CPU is also supported by setting device to 'cpu'.python test_sketch.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/anything_sketch/human.png --ckpt models/anything-v4.0-pruned.ckpt --type_in image
python gradio_sketch.py
- Keypose to Image Generation
python test_keypose.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/keypose/iron.png --type_in pose
- Image to Image Generation
python test_keypose.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/sketch/human.png --type_in image
- Generation with Anything setting
You can use gradio to experience all these three functions at once. CPU is also supported by setting device to 'cpu'.python test_keypose.py --plms --auto_resume --prompt "A beautiful girl" --path_cond examples/sketch/human.png --ckpt models/anything-v4.0-pruned.ckpt --type_in image
python gradio_keypose.py
python test_seg.py --plms --auto_resume --prompt "A black Honda motorcycle parked in front of a garage" --path_cond examples/seg/motor.png
python test_seg_sketch.py --plms --auto_resume --prompt "An all white kitchen with an electric stovetop" --path_cond examples/seg_sketch/mask.png --path_cond2 examples/seg_sketch/edge.png
python test_sketch_edit.py --plms --auto_resume --prompt "A white cat" --path_cond examples/edit_cat/edge_2.png --path_x0 examples/edit_cat/im.png --path_mask examples/edit_cat/mask.png
The following is the detailed structure of a Stable Diffusion model with the T2I-Adapter.
The corresponding edge maps are predicted by PiDiNet. The sketch T2I-Adapter can well generalize to other similar sketch types, for example, sketches from the Internet and user scribbles.
The keypose results predicted by the MMPose. With the keypose guidance, the keypose T2I-Adapter can also help to generate animals with the same keypose, for example, pandas and tigers.
Once the T2I-Adapter is trained, it can act as a plug-and-play module and can be seamlessly integrated into the finetuned diffusion models without re-training, for example, Anything-4.0.
When combined with the inpaiting mode of Stable Diffusion, we can realize local editing with user specific guidance.
Adapter can be used to enhance the SD ability to combine different concepts.
We can realize the sequential editing with the adapter guidance.
Stable Diffusion results guided with the segmentation and sketch adapters together.