Code Monkey home page Code Monkey logo

mulan's Introduction

🌻 MuLan

PyPI - Downloads

# pip install mulankit
from diffusers import StableDiffusionPipeline
+ import mulankit

pipe = StableDiffusionPipeline.from_pretrained('Lykon/dreamshaper-8')
+ pipe = mulankit.transform(pipe, 'mulanai/mulan-lang-adapter::sd15_aesthetic.pth')
image = pipe('一只蓝色的🐶 in the 바다').images[0]
一只蓝色的 🐶 in the 바다 (Dreamshaper-8) レゴシュワルツェネッガー (SDXL-lightning) 一只可爱的猫头鹰 (MVDream) 海浪风景 (AnimateDiff)
dreamshaper8 一只戴着帽子的 rabbit レゴアーノルド・シュワルツェネッガー 海浪

What is it ?

We present MuLan, a versatile framework to equip any diffusion model with multilingual generation abilities natively by up to 110+ languages around the world. With properly trained text encoder from noisy data, we demonstrate that MuLan could be trained on English only data and support other languages zero-shot. Additionally, we introduce Language Adapter. A language adapter with less than 20M parameters, trained against a frozen denoiser and a text encoder, can be readily combined with any homologous community models/tools, such as LoRA, LCM, ControlNet, and IP-Adapter, without any finetuning.

MuLan(木兰)可以使任何扩散模型原生地支持多达110多种语言的图像/视频/3D生成能力。通过使用带噪声的海量数据适当训练的文本编码器,我们展示了MuLan可以仅在英语数据上进行训练并且支持其他语言的零样本生成。此外,我们引入了语言适配器。一个具有不到20M参数的简单映射网络,在一个冻结的去噪器和文本编码器上训练,即可无需任何微调地与任何同类社区模型/工具(如LoRA、LCM、ControlNet和IP-Adapter)无缝结合。

News

  • release technical report
  • 2024-5-14: release code and models

How to use

MuLan supports

  • Base models: Stable Diffusion 1.5, 2.1, XL, Pixart-Alpha/Sigma.
  • Downstream models: ControlNet, LCM, LoRA, finetuned models and etc.
  • Video models: AnimateDiff.
  • 3D models: MVDream.

Please refer to the USAGE.md and examples for more details.

Model Release

Model Description Link
MuLan-Language-Adapter Adapters for SDXL, SD1.5/2.1, Pixart hf-model
MuLan-Pixart Full finetuned model hf-model

See more at our Huggingface 🌻 Homepage.

Citation

If you find this repo helpful, please considering citing us.

@article{lai2024mulan,
  title={MuLan: Adapting Multilingual Diffusion Models for 110 + Languages},
  year={2024}
}

Visitors

Acknowledgement

Our work is made possible by the open-source of these great works.

Stable Diffusion · Pixart-Alpha · InternVL

mulan's People

Contributors

zeqiang-lai avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.