highcwu / controllora Goto Github PK

View Code? Open in Web Editor NEW

552.0 552.0 27.0 9.2 MB

ControlLoRA: A Lightweight Neural Network To Control Stable Diffusion Spatial Information

License: Apache License 2.0

Python 99.96% Shell 0.04%

controllora's People

Contributors

Stargazers

Watchers

controllora's Issues

combine different conditions

Hi @HighCWu ,

Thanks for sharing the great work.

Does ControlLoRA support combine multiple conditional inputs during inference?

For example, I independently trained

one canny edge map ControlLoRA model
one semantic segmentation map ControlLoRA model

During inference, can I use the edge map and segmentation map simultaneously to guide the generation process?

what about the speed

Is control-lora more quickly than original control-net?

How to use controlNet with pretrained LoRA?

I want to combine the pretrained LORA_model and pretrained controlnet_mdoel like "[control_sd15_depth.pth]",and export new model.

How to realize this function?

Are you using your controllora on a Lora model or a full model?

Your output images are low in quality, is this because you are using a Lora trained stable diffusion model or are you using a full model?

May I ask if it actually trains a ControlNet and lora?

Training the lora seems to require only a small amount of data, but the data you are using is very large, I guess it essentially involves training a ControlNet and then a LoRA right? (Please correct me if my understanding is wrong, thanks!)

Try start train on CPU Intel

Hello

I tried to run train_fill50k.py, but I get this error. I removed --mixed_precision="fp16" from the launch command and got this result. Running the original command on the GPU in the collab also returns this result.

05/08/2023 14:04:05 - INFO - __main__ - ***** Running training *****
05/08/2023 14:04:05 - INFO - __main__ -   Num examples = 50000
05/08/2023 14:04:05 - INFO - __main__ -   Num Epochs = 100
05/08/2023 14:04:05 - INFO - __main__ -   Instantaneous batch size per device = 1
05/08/2023 14:04:05 - INFO - __main__ -   Total train batch size (w. parallel, distributed & accumulation) = 1
05/08/2023 14:04:05 - INFO - __main__ -   Gradient Accumulation steps = 1
05/08/2023 14:04:05 - INFO - __main__ -   Total optimization steps = 5000000
Steps:   0%|          | 0/5000000 [00:00<?, ?it/s]
Steps:   0%|          | 0/5000000 [00:00<?, ?it/s]/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_ddpm.py:172: FutureWarning: Accessing `num_train_timesteps` directly via scheduler.num_train_timesteps is deprecated. Please use `  instead`
  deprecate(
Traceback (most recent call last):
  File "/Users/petro/PycharmProjects/ControlLoRA/train_text_to_image_control_lora.py", line 1006, in <module>
    main()
  File "/Users/petro/PycharmProjects/ControlLoRA/train_text_to_image_control_lora.py", line 782, in main
    model_pred = unet(noisy_latents, timesteps, encoder_hidden_states).sample
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/diffusers/models/unet_2d_condition.py", line 695, in forward
    sample, res_samples = downsample_block(
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/diffusers/models/unet_2d_blocks.py", line 867, in forward
    hidden_states = attn(
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/diffusers/models/transformer_2d.py", line 265, in forward
    hidden_states = block(
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/diffusers/models/attention.py", line 294, in forward
    attn_output = self.attn1(
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/diffusers/models/attention_processor.py", line 243, in forward
    return self.processor(
  File "/Users/petro/PycharmProjects/ControlLoRA/models.py", line 230, in __call__
    attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length)
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/diffusers/models/attention_processor.py", line 302, in prepare_attention_mask
    deprecate(
  File "/Users/petro/PycharmProjects/ControlLoRA/venv/lib/python3.9/site-packages/diffusers/utils/deprecation_utils.py", line 18, in deprecate
    raise ValueError(
ValueError: The deprecation tuple ('batch_size=None', '0.0.15', 'Not passing the `batch_size` parameter to `prepare_attention_mask` can lead to incorrect attention mask preparation and is deprecated behavior. Please make sure to pass `batch_size` to `prepare_attention_mask` when preparing the attention_mask.') should be removed since diffusers' version 0.15.0 is >= 0.0.15

The example in the readme page seems to be bad (not as good as origin controlnet model result)

The example in the readme page seems to be bad (not as good as origin controlnet model result).
Why?

Can controllora be used to learn to make templates for loras?

Can controlloras be used to generate a template, of let's say a pose for a lora?

By example, can I train a controllora on rpg maker sprites, and eventually get something like a rpg maker lora that can sucesfully control a character and style loras to generate rpg maker sprites, sucesfully?

Or do controlloras have a diferent planned use.
Many thanks.

The use of "test_dreambooth_lora.py" and "test_text_to_image_control_lora.py"

Hi thanks for great work! I noticed you added some new files relating to dreambooth_lora and text_to_image_control_lora. May I ask what are they used for and how to use them? Could you please provide some examples? Thanks!

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.