Switch to SD v1.5 about paint-with-words-sd HOT 4 CLOSED

I made a simple colab example just now, you can try it out here: paint-with-words colab

Originally posted by @shreydan in #4 (comment)

You can confirm the above hf_model_path change on the colab mentioned here by editing the cell that has the img = paint_with_words( (first cell after the 'Run Model' heading)

We can also explicitly set the seed parameter here as well, to make comparisons between the 2 models easier.

Original version using SD v1.4

New version using SD v1.5

from paint-with-words-sd.

This is documented in the README already:

• https://github.com/cloneofsimo/paint-with-words-sd#using-other-fine-tuned-models

You can see that pww_load_tools takes a hf_model_path and/or local_model_path here:

pww_load_tools is called by paint_with_words, which can also be passed hf_model_path and/or local_model_path, with hf_model_path defaulting to "CompVis/stable-diffusion-v1-4"

So it looks like it may be even easier than the README suggests, and you should just be able to pass "runwayml/stable-diffusion-v1-5" directly into the hf_model_path parameter of paint_with_words as it's called in the 'Basic Usage' section:

• https://github.com/cloneofsimo/paint-with-words-sd#basic-usage

Something like this would presumably work:

img = paint_with_words(
    color_context=color_context,
    color_map_image=color_map_image,
    input_prompt=input_prompt,
    num_inference_steps=30,
    guidance_scale=7.5,
    device="cuda:0",
+   hf_model_path="runwayml/stable-diffusion-v1-5"
)

from paint-with-words-sd.

You can see that pww_load_tools takes a hf_model_path and/or local_model_path here:

One advantage of this method that you can also change the scheduler, which might be desirable for performance or quality issues. (I found recently that the model I'm using often has blatant graphical issues even with a large step count on LMS, but EulerAncestralDiscreteScheduler or EulerDiscreteScheduler do much better. Just don't forget to import your scheduler of choice from diffusers. (Something the example currently doesn't show.)

from paint-with-words-sd.

Nice note @hiss-remi , ill add this feature as well.

from paint-with-words-sd.