Code Monkey home page Code Monkey logo

res-adapter's Introduction

ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models

Jiaxiang Cheng, Pan Xie*, Xin Xia, Jiashi Li, Jie Wu, Yuxi Ren, Huixia Li, Xuefeng Xiao, Min Zheng, Lean Fu (*Corresponding author)

AutoML, ByteDance Inc.

โญ If ResAdapter is helpful to your images or projects, please help star this repo. Thanks! ๐Ÿค—

GitHub Org's stars

Replicate Hugging Face ComfyUI visitors

We propose ResAdapter, a plug-and-play resolution adapter for enabling any diffusion model generate resolution-free images: no additional training, no additional inference and no style transfer.

Comparison examples between resadapter and dreamlike-diffusion-1.0.

Release

Quicktour

We provide a standalone example code to help you quickly use resadapter with diffusion models.

Comparison examples (640x384) between resadapter and dreamshaper-xl-1.0. Top: with resadapter. Bottom: without resadapter.

# pip install diffusers, transformers, accelerate, safetensors, huggingface_hub
import torch
from torchvision.utils import save_image
from safetensors.torch import load_file
from huggingface_hub import hf_hub_download
from diffusers import AutoPipelineForText2Image, DPMSolverMultistepScheduler

generator = torch.manual_seed(0)
prompt = "portrait photo of muscular bearded guy in a worn mech suit, light bokeh, intricate, steel metal, elegant, sharp focus, soft lighting, vibrant colors"
width, height = 640, 384

# Load baseline pipe
model_name = "lykon-models/dreamshaper-xl-1-0"
pipe = AutoPipelineForText2Image.from_pretrained(model_name, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas=True, algorithm_type="sde-dpmsolver++")

# Inference baseline pipe
image = pipe(prompt, width=width, height=height, num_inference_steps=25, num_images_per_prompt=4, output_type="pt").images
save_image(image, f"image_baseline.png", normalize=True, padding=0)

# Load resadapter for baseline
resadapter_model_name = "resadapter_v1_sdxl"
pipe.load_lora_weights(
    hf_hub_download(repo_id="jiaxiangc/res-adapter", subfolder=resadapter_model_name, filename="pytorch_lora_weights.safetensors"), 
    adapter_name="res_adapter",
    ) # load lora weights
pipe.set_adapters(["res_adapter"], adapter_weights=[1.0])
pipe.unet.load_state_dict(
    load_file(hf_hub_download(repo_id="jiaxiangc/res-adapter", subfolder=resadapter_model_name, filename="diffusion_pytorch_model.safetensors")),
    strict=False,
    ) # load norm weights

# Inference resadapter pipe
image = pipe(prompt, width=width, height=height, num_inference_steps=25, num_images_per_prompt=4, output_type="pt").images
save_image(image, f"image_resadapter.png", normalize=True, padding=0)

Download

Models

We have released all resadapter weights, you can download resadapter models from Huggingface. The following is our resadapter model card:

Models Parameters Resolution Range Ratio Range Links
resadapter_v1_sd1.5 0.9M 128 <= x <= 1024 0.5 <= r <= 2 Download
resadapter_v1_sd1.5_extrapolation 0.9M 512 <= x <= 1024 0.5 <= r <= 2 Download
resadapter_v1_sd1.5_interpolation 0.8M 128 <= x <= 512 0.5 <= r <= 2 Download
resadapter_v1_sdxl 0.5M 256 <= x <= 1536 0.5 <= r <= 2 Download
resadapter_v1_sdxl_extrapolation 0.5M 1024 <= x <= 1536 0.5 <= r <= 2 Download
resadapter_v1_sdxl_interpolation 0.4M 256 <= x <= 1024 0.5 <= r <= 2 Download

Hint1: We update the resadapter name format according to controlnet.

Hint2: If you want use resadapter with personalized diffusion models, you should download them from CivitAI.

Hint3: If you want use resadapter with ip-adapter, controlnet and lcm-lora, you should download them from Huggingface.

Hint4: Here is an installation guidance for preparing environment and downloading models.

Inference

If you want generate images in our inference script, you should download related models and fill in configs. Then you can directly run this script.

python main.py --config /path/to/file

ResAdapter with Personalized Models for Text to Image

Comparison examples (960x1104) between resadapter and dreamshaper-7. Top: with resadapter. Bottom: without resadapter.

ResAdapter with ControlNet for Image to Image

Comparison examples (840x1264) between resadapter and lllyasviel/sd-controlnet-canny. Top: with resadapter, bottom: without resadapter.

ResAdapter with ControlNet-XL for Image to Image

Comparison examples (336x504) between resadapter and diffusers/controlnet-canny-sdxl-1.0. Top: with resadapter, bottom: without resadapter.

ResAdapter with IP-Adapter for Face Variance

Comparison examples (864x1024) between resadapter and h94/IP-Adapter. Top: with resadapter, bottom: without resadapter.

ResAdapter with LCM-LoRA for Speeding up

Comparison examples (512x512) between resadapter and dreamshaper-xl-1.0 with lcm-sdxl-lora. Top: with resadapter, bottom: without resadapter.

Community Resource

Gradio

ComfyUI

helper_video_720p.mp4

Local Gradio Demo

Run the following script:

python app.py

Usage Tips

  • We recommend users to use interpolation version to generate lower-resolution images.
  • We recommend users to use extrapolation version to generate higher-resolution images.
  • We recommend users to use resadapter_v1_sd1.5 and resadapter_v1_sdxl for deploying resadapter to generate images with broader resolution.
  • We strongly recommend that you use the prompt corresponding to the personalized model, which helps to enhance the quality of the image.

Acknowledgements

  • ResAdapter is developed by AutoML Team at ByteDance Inc, all copyright reserved.
  • Thanks to the HuggingFace gradio team for their free GPU support!
  • Thanks to the IP-Adapter, ControlNet, LCM-LoRA for their nice work.

๐ŸŒŸ Star History

Star History Chart

โœˆ๏ธ Citation

If you find ResAdapter useful for your research and applications, please cite us using this BibTeX:

@article{cheng2024resadapter,
  title={ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models},
  author={Cheng, Jiaxiang and Xie, Pan and Xia, Xin and Li, Jiashi and Wu, Jie and Ren, Yuxi and Li, Huixia and Xiao, Xuefeng and Zheng, Min and Fu, Lean},
  booktitle={arXiv preprint arxiv:2403.02084},
  year={2024}
}

For any question, please feel free to contact us via [email protected] or [email protected].

res-adapter's People

Contributors

jiaxiangc avatar chenxwh avatar ameerazam08 avatar fengyuzzz avatar zweifisch avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.