Possibility to use with byt5-small ? about paella HOT 6 OPEN

Unfortunately, Paella requires at least 30GB of memory since we trained on big conditoning models. We will change that in future models.

from paella.

So not even on RTX 3090? That's a pretty huge restriction if I do say so

from paella.

Definitely. Paella is not a finished research project and we are still working on improving on many things. One thing that you can do is just not load t5 at all and enable t5=False in the sampling script. Then you can also do del clip.visual and then it should work probably. Let me know if that works for you!

from paella.

I tried but I think there are references to t5 stuff even outside the checks for t5, and there seem to be more complications in the Paella model with the byt5_mapper

from paella.

I managed to use Paella on colab using ~10GB of VRAM by simply using the CLIP image model to create variations, and deleting the T5 model and the prior model (I could also delete the text clip encoder).

If you have a rtx 3090 I think you can try loading the byT5-XL model alone, save the embeddings, delete the model and load the rest (if it doesn't work, try loading the model in half precision).

from paella.

I managed to use Paella on colab using ~10GB of VRAM by simply using the CLIP image model to create variations, and deleting the T5 model and the prior model (I could also delete the text clip encoder).

If you have a rtx 3090 I think you can try loading the byT5-XL model alone, save the embeddings, delete the model and load the rest (if it doesn't work, try loading the model in half precision).

Can you share your code for image variations? Thank you.

from paella.