irisrainbowneko / dreamartist-stable-diffusion Goto Github PK

The example embeddings zip is missing the real-dog embeddings. Are these still available somewhere?

My rig has a 4090 and I'm getting a error about CPU?
same one as #16
so not sure if duplicate or not?

Loaded a total of 2 words embeddings.
Embeddings: Elins-neg, Elins
Training at rate of 0.005 until step 100000
Preparing dataset...
100%|██████████████████████████████████████████████████████████████████████████████████| 12/12 [00:12<00:00, 1.07s/it]
0%| | 0/100000 [00:03<?, ?it/s]
Applying xformers cross attention optimization.
Error completing request
Arguments: ('Elins', '0.005', 1, 'C:\Users\PERSON\Pictures\ELINS', 'dream_artist', 768, 768, 100000, 250, 250, 'E:\DreamArtist-stable-diffusion\textual_inversion_templates\style_filewords.txt', True, False, '', '', 20, 0, 7, -1.0, 512, 512, 5.0, '', True, True, 1, 1, 1.0, 25.0, 1.0, 25.0, 0.9, 0.999) {}
Traceback (most recent call last):
File "E:\DreamArtist-stable-diffusion\modules\ui.py", line 185, in f
res = list(func(*args, **kwargs))
File "E:\DreamArtist-stable-diffusion\webui.py", line 54, in f
res = func(*args, **kwargs)
File "E:\DreamArtist-stable-diffusion\modules\dream_artist\ui.py", line 36, in train_embedding
embedding, filename = modules.dream_artist.cptuning.train_embedding(*args)
File "E:\DreamArtist-stable-diffusion\modules\dream_artist\cptuning.py", line 436, in train_embedding
output = shared.sd_model(x, c_in, scale=cfg_scale)
File "E:\DreamArtist-stable-diffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "E:\DreamArtist-stable-diffusion\repositories\stable-diffusion\ldm\models\diffusion\ddpm.py", line 879, in forward
return self.p_losses(x, c, t, *args, **kwargs)
File "E:\DreamArtist-stable-diffusion\modules\dream_artist\cptuning.py", line 286, in p_losses_hook
logvar_t = self.logvar[t].to(self.device)
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

I'm not too sure how to get the negative prompt through besides txt2img which only generates previews.
I tried to follow the same pattern than with generation parameters txt fiels coming along generated images:

prompt text
Negative prompt: prompt text

I'm not sure if the negative prompt is getting through or not.

As there's a checkbox saying "positive filewords only", I assume there's such a thing as negative filewords. So I try to tinker around but there's not any information online about that subject.

我clone repo之后，把models文件夹复制进去，不管是txt2img还是img2img，跑出来的都是噪点图
中途还报了缺少sklearn模块的错误，不过我自己解决了，不知道有没有影响

最新版的webui支持插件形式了，可以考虑做成插件？这样不需要merge，可以立即使用master的最新功能，并且项目能减小很多，可维护性更好。

The advantages of making this an extension instead of a standalone repo/branch:

1. no merging require.
2. keep up with the latest webui/master.
3. smaller code base.

Followed the install instructions but the DreamArtist create embedding tab is not appearing, it just looks like the vanilla ui with the regular train tab.

Simply speaking for vanilla Stable Diffusion, during training, given a x(0), the first stage model encodes it into latent z(0).
After adding noise eps(t), we get nosiy z(t).
The vanilla Stable Diffusion calculates the loss between eps(t) and the UNet-predicted noisy eps(t)_pred for timestep t.
Using eps(t)_pred, we denoise z(t) and get z(t-1)_pred.

Your paper proposes a reconstruction constraint loss, which calculates the L1 loss between ground-truth x(0) and the decoded predicted z(0)_pred like:
$$||x(0) - D(z_{pred}(0))||_1$$

However, obtaining z(0) from z(t-1)_pred requires t-1 denoising steps, which seems very time-cost during training (forward called by t-1 times for an input batch).
My question is: Do you implement the reconstruction loss by denoising for t-1 steps during training? Or are you using a more efficient method to get z(0)_pred from z(t-1)_pred?

ive had to redownload everything after formatting and noticed that both ptun-futa and ptun-futa-uc got removed from the current pre-trained embeddings pack in releases, any way to get those 2 back?

请问下，仅支持novelAI还是原生V1.4model也可以的？

Hey bro/sister, i really like the embeddings you have included in this, so i thought, why not post them on https://civitai.com , i posted myself the nahida one, https://civitai.com/models/4062/nahida-genshin-impact
if you can publish the embeddings to this site yourself so i can remove the one published by me, you could make dreamartist more famous as well as make many people happy, (people like me who love genshin impact).
I know you technically posted the embeddings on github, and so anyone can use them to do whatever they want (not really, but roughly), but.. if you don't want them to be posted anywhere else, or you want me to remove what i already posted (nahida) from civitai, let me know (I don't know how to contact you, so i will submit an issue xd)

Hi.
I tried dreamartist, but I get a GPU out of memory error.
The error code is below.
Can you please advise me?

Python 3.10.7 (tags/v3.10.7:6cc6b13, Sep 5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)]

Training at rate of 0.005 until step 3000
Preparing dataset...
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.99s/it]
0%| | 0/3000 [00:03<?, ?it/s]
Applying cross attention optimization (Doggettx).
Error completing request
Arguments: ('test1', '0.005', 1, 'D:\Program Files\stable-diffusion-webui-dream\extensions\DreamArtist\imgs\train', 'textual_inversion', 512, 512, 3000, 500, 500, 'D:\Program Files\stable-diffusion-webui-dream\textual_inversion_templates\style_filewords.txt', True, False, '', '', 20, 0, 7, -1.0, 512, 512, 5.0, '', True, False, 1, 1) {}
Traceback (most recent call last):
File "D:\Program Files\stable-diffusion-webui-dream\modules\ui.py", line 185, in f
res = list(func(*args, **kwargs))
File "D:\Program Files\stable-diffusion-webui-dream\webui.py", line 54, in f
res = func(*args, **kwargs)
File "D:\Program Files\stable-diffusion-webui-dream\extensions\DreamArtist\scripts\dream_artist\ui.py", line 30, in train_embedding
embedding, filename = dream_artist.cptuning.train_embedding(*args)
File "D:\Program Files\stable-diffusion-webui-dream\extensions\DreamArtist\scripts\dream_artist\cptuning.py", line 430, in train_embedding
loss.backward()
File "D:\Program Files\stable-diffusion-webui-dream\venv\lib\site-packages\torch_tensor.py", line 396, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "D:\Program Files\stable-diffusion-webui-dream\venv\lib\site-packages\torch\autograd_init_.py", line 173, in backward
Variable.execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "D:\Program Files\stable-diffusion-webui-dream\venv\lib\site-packages\torch\autograd\function.py", line 253, in apply
return user_fn(self, *args)
File "D:\Program Files\stable-diffusion-webui-dream\repositories\stable-diffusion\ldm\modules\diffusionmodules\util.py", line 139, in backward
input_grads = torch.autograd.grad(
File "D:\Program Files\stable-diffusion-webui-dream\venv\lib\site-packages\torch\autograd_init.py", line 276, in grad
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 8.00 GiB total capacity; 5.31 GiB already allocated; 0 bytes free; 6.71 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

hello! tested on 3090 ti gpu-i get better results with dreambooth in 10 minutes than with this in one hour. any advise for training, naming or something? best wishes

Hi! I am working on a pr for this for diffusers by huggingface and I wondered if anyone has some idea on why the 1e-3 noise is getting added to the negative initial embeddings. Also, am I correct in thinking that the negative initial token should be chosen differently from the positive initial token?

install the extension but i can't write anything , only name

I see there is a embedding.zip in the release page. The file names inside are mysterious, I cannot tell which embedding is for what. Do you have a list of them? Thanks

The link you included in your pull request seemed to lead to a deleted video. Do you have any blog/video explanation for how this works?

I can't seem to run DreamArtist in a fresh install because of this error, and I can't seem to find out where did it went wrong. Got any ideas on how to fix this? Thanks in advance.

Traceback (most recent call last):
  File "/content/stable-diffusion-webui/modules/call_queue.py", line 45, in f
    res = list(func(*args, **kwargs))
  File "/content/stable-diffusion-webui/modules/call_queue.py", line 28, in f
    res = func(*args, **kwargs)
  File "/content/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/ui.py", line 30, in train_embedding
    embedding, filename = dream_artist.cptuning.train_embedding(*args)
  File "/content/stable-diffusion-webui/extensions/DreamArtist-sd-webui-extension/scripts/dream_artist/cptuning.py", line 542, in train_embedding
    loss.backward()
  File "/usr/local/envs/automatic/lib/python3.10/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/envs/automatic/lib/python3.10/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper__convolution_backward)

As title. It seems there is no any response after the button is pressed.

The newest commission is:
AUTOMATIC1111/stable-diffusion-webui@38b7186

Traceback (most recent call last):
  File "G:\workspace\DreamArtist-stable-diffusion\modules\ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "G:\workspace\DreamArtist-stable-diffusion\webui.py", line 54, in f
    res = func(*args, **kwargs)
  File "G:\workspace\DreamArtist-stable-diffusion\modules\dream_artist\ui.py", line 36, in train_embedding
    embedding, filename = modules.dream_artist.cptuning.train_embedding(*args)
  File "G:\workspace\DreamArtist-stable-diffusion\modules\dream_artist\cptuning.py", line 436, in train_embedding
    output = shared.sd_model(x, c_in, scale=cfg_scale)
  File "C:\Users\unknown\miniconda3\envs\pytorch-1.13\lib\site-packages\torch\nn\modules\module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "G:\workspace\DreamArtist-stable-diffusion\repositories\stable-diffusion\ldm\models\diffusion\ddpm.py", line 879, in forward
    return self.p_losses(x, c, t, *args, **kwargs)
  File "G:\workspace\DreamArtist-stable-diffusion\modules\dream_artist\cptuning.py", line 286, in p_losses_hook
    logvar_t = self.logvar[t].to(self.device)
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

Do you know why this is happening? I can fix this by changing that line to logvar_t = self.logvar.to(self.device)[t] but I don't know why self.logvar is not moved to GPU.

Tried many times training, but still no good result
Like in the first ~400steps it seems like it's going somewhere with a loss ~0.3
but after that, the loss suddenly shrink to ~0.05 and become stagnated
also I don't know how to use both the convnext model?
tried lr from 0.01-0.0025 cfg 3-5 without success
Is there something I've done wrong?
Using Anything-V3.0-pruned.ckpt, setting:
{
"bs": 1,
"cfg": 3.0,
"lr": "0.0025",
"neg": true,
"neg_lr_w": 1,
"prompt_len": [
6,
768
],
"prompt_len_neg": [
12,
768
],
"rec": true,
"rec_loss_w": 1,
"size": 384
}
Input image:

Output after 5000 steps:

prompt_tuning_loss.csv

I tried to tweak the notebook intended for original webui (https://colab.research.google.com/drive/1gk-R0lx0T_Pk3kETvcMDs4Js8vxzSaKm?usp=sharing#scrollTo=mEvcDwRpJfxM) to work for this
Basically only altered the folder name and change the model from nai to anything
But it failed at the launching stage

Why was I linked to a "Stand alone" version, but then it tells me to install automatic? And the other one is an extension for webui, which is what automatics is? I'm so confused. The instructions for installing aren't clear either. I have automatic installed, and I cloned this, so what next?

Training at rate of 0.003 until step 8000
Preparing dataset...
100%|████████████████████████████████████████████████████████████████████████████████| 160/160 [00:01<00:00, 80.21it/s]
[Epoch 1: 0/1]loss: 0.0001802, grad:0.0000073, grad_neg:0.0000021: 0%| | 0/7999 [00:00<?, ?it/s]
Applying attention optimization: Doggettx... done.
*** Error completing request
*** Arguments: ('task(t6txrwmjoy64bsg)', 'style2', 114514, '0.003', 1, 'C:\Users\tomwe\Desktop\iconex', 'dream_artist', 512, 512, 8000, 2, 500, 'C:\Users\tomwe\sd\_textual_inversion_templates\style_filewords.txt', False, False, 'masterpiece,best quality,style2', 'style2-neg,bad anatomy,bad hands,text,error,missing fingers,extra digit,fewer digits,cropped,worst quality,low quality,normal quality,jpeg artifacts,signature,watermark,username,blurry,blurred', 20, 0, 7, -1.0, 512, 512, '5.0', '', True, True, False, 1, 1, 0.97, 25.0, 1.0, 25.0, 0.9, 0.999, False, 1, False, '0.000005') {}
Traceback (most recent call last):
File "C:\Users\tomwe\sd\modules\call_queue.py", line 58, in f
res = list(func(*args, **kwargs))
File "C:\Users\tomwe\sd\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
File "C:\Users\tomwe\sd\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\ui.py", line 32, in train_embedding
embedding, filename = dream_artist.cptuning.train_embedding(*args)
File "C:\Users\tomwe\sd\extensions\DreamArtist-sd-webui-extension\scripts\dream_artist\cptuning.py", line 646, in train_embedding
processed = processing.process_images(p)
File "C:\Users\tomwe\sd\extensions\sd-webui-prompt-history\lib_history\image_process_hijacker.py", line 21, in process_images
res = original_function(p)
File "C:\Users\tomwe\sd\modules\processing.py", line 677, in process_images
res = process_images_inner(p)
File "C:\Users\tomwe\sd\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
File "C:\Users\tomwe\sd\modules\processing.py", line 794, in process_images_inner
samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
File "C:\Users\tomwe\sd\modules\processing.py", line 1046, in sample
self.sampler = sd_samplers.create_sampler(self.sampler_name, self.sd_model)
File "C:\Users\tomwe\sd\extensions\sd-webui-neutral-prompt\lib_neutral_prompt\cfg_denoiser_hijack.py", line 254, in create_sampler_hijack
if name.startswith(('DDIM', 'PLMS', 'UniPC')):
AttributeError: 'NoneType' object has no attribute 'startswith'

hello,did you implement the dreambooth yourself?

Previously i succeed doing training on google colab.
But recently this shows up when i press the train button.

venv "E:\sd-APT-stable-diffusion-auto-prompt-master\venv\Scripts\Python.exe"
Python 3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)]
Commit hash:
Installing requirements for Web UI
Launching Web UI with arguments:
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Traceback (most recent call last):
File "E:\sd-APT-stable-diffusion-auto-prompt-master\launch.py", line 152, in
start_webui()
File "E:\sd-APT-stable-diffusion-auto-prompt-master\launch.py", line 148, in start_webui
import webui
File "E:\sd-APT-stable-diffusion-auto-prompt-master\webui.py", line 79, in
shared.sd_model = modules.sd_models.load_model()
File "E:\sd-APT-stable-diffusion-auto-prompt-master\modules\sd_models.py", line 154, in load_model
sd_model = instantiate_from_config(sd_config.model)
File "E:\sd-APT-stable-diffusion-auto-prompt-master\repositories\stable-diffusion\ldm\util.py", line 85, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "E:\sd-APT-stable-diffusion-auto-prompt-master\repositories\stable-diffusion\ldm\models\diffusion\ddpm.py", line 461, in init
self.instantiate_cond_stage(cond_stage_config)
File "E:\sd-APT-stable-diffusion-auto-prompt-master\repositories\stable-diffusion\ldm\models\diffusion\ddpm.py", line 519, in instantiate_cond_stage
model = instantiate_from_config(config)
File "E:\sd-APT-stable-diffusion-auto-prompt-master\repositories\stable-diffusion\ldm\util.py", line 85, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "E:\sd-APT-stable-diffusion-auto-prompt-master\repositories\stable-diffusion\ldm\modules\encoders\modules.py", line 141, in init
self.tokenizer = CLIPTokenizer.from_pretrained(version, local_files_only=True)
File "E:\sd-APT-stable-diffusion-auto-prompt-master\venv\lib\site-packages\transformers\tokenization_utils_base.py", line 1768, in from_pretrained
raise EnvironmentError(
OSError: Can't load tokenizer for './models/clip-vit-large-patch14'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure './models/clip-vit-large-patch14' is the correct path to a directory containing all relevant files for a CLIPTokenizer tokenizer.
请按任意键继续. . .

Everyone I asked on the SD discord said the results looked nothing like the training image. The results in the readme look great so I'm left wondering what went wrong? Did they use a bad learning rate? step count? Is there something weird with sample images that doesn't apply when using the embeddings in a prompt normally? I really want to try this, but I fear it will be a waste of time if I don't know the right way to use it

When I train with a picture from the paper

my training results are as follows with the prompt "a painting of city, art by ch_fg4; ch_fg-neg" :

but the results in the paper are like this:

My training parameters are as follows:

Is there something wrong with my training process causing this poor result？

Thanks！

how many steps are needed for good results

after installing it.

Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Traceback (most recent call last):
  File "Z:\__TESTS\DreamArtist-stable-diffusion\launch.py", line 252, in <module>
    start()
  File "Z:\__TESTS\DreamArtist-stable-diffusion\launch.py", line 247, in start
    webui.webui()
  File "Z:\__TESTS\DreamArtist-stable-diffusion\webui.py", line 173, in webui
    app.add_middleware(GZipMiddleware, minimum_size=1000)
  File "Z:\__TESTS\DreamArtist-stable-diffusion\venv\lib\site-packages\starlette\applications.py", line 139, in add_middleware
    raise RuntimeError("Cannot add middleware after an application has started")
RuntimeError: Cannot add middleware after an application has started
Press any key to continue . . .