trouble shooting `woman` generations? about dreambooth-ion-cannon HOT 4 CLOSED

I can confirm I get absolutely identical images on a second run on a clean new instance. so it seems somehow the training class is being ignored. the target_name "emma" is in the file names tho:

emma_blackwhite_0.png

from dreambooth-ion-cannon.

Oh it's a GPU out of mem error.
also noticed i uploaded a .DS_Store mac file by accident...

I wonder if you have guidelines on how many images could be uploaded generally?
eg how many / big files?
seems to be a problem for the Tesla instances, even the a 10x Tesla seems capped at 16GB of VRAM:

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.78 GiB total capacity; 14.49 GiB already allocated; 15.69 MiB free; 14.63 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

from dreambooth-ion-cannon.

Must have 24GB - "One critical choice you need to make is that whatever you rent must have 24 GB VRAM"

from dreambooth-ion-cannon.

got it thanks! i overlooked that in the README. might be worth having some code to do a startup check at the beginning and fatal if there's for sure not enough vram.

from dreambooth-ion-cannon.