missing options in Dreambooth training about kohya_ss HOT 17 OPEN

from kohya_ss.

@b-fission Does TE1 and TE2 learning rates only apply to Dreambooth? Wanted to try it in Lora module

Yes, TE1 and TE2 learning rates are for Dreambooth. Lora training only has a single TE learning rate option in the gui.

But I wonder if TE1 and TE2 rates can be specified manually under "Additional parameters" for Lora if one were inclined to try.

from kohya_ss.

The option for "Stop TE (% of total steps)" only seems to be relevant in Dreambooth for SD 1.x or 2.0, but not SDXL.

from kohya_ss.

I just checked and when I select an SDXL and specify it as being SDXL the options are showing:

As far as the 2nd one I will need to do more research... but this is the 1st time it is reported... so maybe the issue is linked to the SDXL option selection for the model?

from kohya_ss.

It seems the entire "SDXL Specific Parameters" group is missing in the master branch (and release) versions. I even searched the source code for "fused backward pass" and got nothing.

When I switched to dev, those options do show up.

from kohya_ss.

There are many fixes in dev. I am waiting for Kohya to merge dev to main to promote the dev branch to master. In the mean time you can use the dev branch as it should be more stable than master at this point.

from kohya_ss.

@bmaltais I am on dev, and also dont see option to checkbox SDXL? So it seems like there can be no training in SDXL at all for the moment. Basically locked out of training. Unless I am missing something

from kohya_ss.

Ok. That might be it, though I loaded my base model from /workspace on runpod
Will check again thx

Comfirming this was my error

from kohya_ss.

Two things on Dreambooth training for SDXL

• Missing checkbox for "No half VAE" leading to NaN errors
• TE1/TE2 learning rates have no effect. The training script indicates it was only training the Unet in SDXL dreambooth. I think the gui isn't passing the arg --train_text_encoder

@b-fission Does TE1 and TE2 learning rates only apply to Dreambooth? Wanted to try it in Lora module

from kohya_ss.

I will try it
Would it be something like
Train_TE1 = true etc..?

from kohya_ss.

That is assuming you want to use a different learning rate for TE1 and TE2.

The args could be entered into Additional parameters as --learning_rate_te1=0.00002 --learning_rate_te2=0.00001 and I've put those rates there as an example. I haven't tried to test Lora training with differing TE learn rates before.

from kohya_ss.

I was thinking of turning Te2 off

By the way for subjects, what worked better for you, dreambooth or Lora implementation?

from kohya_ss.

I think Dreambooth tends to reach the intended results I want with less tweaking compared to Lora.

Afterwards, I can use the Extract Lora utility to generate a Lora from the generated Dreambooth checkpoint, so I can have the flexibility of Lora and save disk space.

from kohya_ss.

I've used both but testing some other optimisers like prodigy and Adafactor besides adamW8bit
Which one do you prefer?

from kohya_ss.

For me, AdamW8bit has worked well enough that I haven't put much time into trying other optimizers.

I've played with Prodigy a few times with Loras and got 'interesting' results, but it sometimes learned too fast and overtrained; probably needs extra setup on my end.

Haven't tried the other ones though. From what little testing I did, Lion8bit and Adafactor are interesting for having lower vram requirements, which makes them viable for training SDXL on mid-range hardware.

from kohya_ss.

thanks for the insights, similar to my understanding. I'll ditch adafactor. Interested mainly in adamw8bit vs prodigy. and testing TE1/TE2. Or TrainTE for some % steps. Though not sure that works still in dreambooth

from kohya_ss.

Unfortunate

from kohya_ss.