yemo-memeda / sd-webui-controlnet Goto Github PK

(WIP) WebUI extension for ControlNet

This extension is for AUTOMATIC1111's Stable Diffusion web UI, allows the Web UI to add ControlNet to the original Stable Diffusion model to generate images. The addition is on-the-fly, the merging is not required.

ControlNet is a neural network structure to control diffusion models by adding extra conditions.

Thanks & Inspired: kohya-ss/sd-webui-additional-networks

• Dragging large file on the Web UI may freeze the entire page. It is better to use the upload file option instead.
• Just like WebUI's hijack, we used some interpolate to accept arbitrary size configure (see scripts/cldm.py)

Some users may need to install the cv2 library before using it: pip install opencv-python

Upgrade gradio if any ui issues occured: pip install gradio==3.16.2

1. Open "Extensions" tab.
2. Open "Install from URL" tab in the tab.
3. Enter URL of this repo to "URL for extension's git repository".
4. Press "Install" button.
5. Reload/Restart Web UI.

1. Put the ControlNet models (.pt, .pth, .ckpt or .safetensors) inside the sd-webui-controlnet/models folder.
2. Open "txt2img" or "img2img" tab, write your prompts.
3. Press "Refresh models" and select the model you want to use. (If nothing appears, try reload/restart the webui)
4. Upload your image and select preprocessor, done.

Currently it supports both full models and trimmed models. Use extract_controlnet.py to extract controlnet from original .pth file.

Pretrained Models: https://huggingface.co/lllyasviel/ControlNet/tree/main/models

Two methods can be used to reduce the model's filesize:

1. Directly extract controlnet from original .pth file using extract_controlnet.py.

2. Transfer control from original checkpoint by making difference using extract_controlnet_diff.py.

All type of models can be correctly recognized and loaded. The results of different extraction methods are discussed in lllyasviel/ControlNet#12 and Mikubill#73.

Pre-extracted model: https://huggingface.co/webui/ControlNet-modules-safetensors

Pre-extracted difference model: https://huggingface.co/kohya-ss/ControlNet-diff-modules

Currently support both sketch Adapter and image Adapter. Note that the impl is experimental, result may differ from original repo. See Adapter Examples for reference.

To use these models:

1. Download files from https://huggingface.co/TencentARC/T2I-Adapter
2. Setup correct config in settings panel - sketch_adapter_v14.yaml for sketch model and image_adapter_v14.yaml for keypose and segmentation model.
3. It's better to use a slightly lower strength (t) when generating images with sketch model, such as 0.6-0.8. (ref: ldm/models/diffusion/plms.py)

• Don't forget to add some negative prompt, default negative prompt in ControlNet repo is "longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality".
• Regarding canvas height/width: they are designed for canvas generation. If you want to upload images directly, you can safely ignore them.

Source	Input	Output
(no preprocessor)
(no preprocessor)

Input	Output

• (Windows) (NVIDIA: Ampere) 4gb - with --xformers enabled, and Low VRAM mode ticked in the UI, goes up to 768x832