Bad Inpainting about stable-diffusion HOT 4 OPEN

Colab and HuggingFace both return the distorted image. You can test this out by using a very low step count like 2 and you will see how the entire image is modified, not just the masked area.

Colab and Huggingface use both the diffusers inpainting pipline. I'm pretty sure, runway does not on their website. I assume, they use the code published in this repo.

So, it's very likely, that the problem is caused by the diffusers inpainting pipeline. I already talked to the diffusers guys. It seems, that this is a known problem and nobody is working on solving it. The diffusers pipeline is imo also the place, where the problem should be fixed.

I solved this problem mostly by reapplying the mask on the generated inpainted image:
final_inpainted = Image.composite(inpainted, image, mask)

This helps for sure in many cases, where there is already an existing object (like the shirt), which should be replaced by a different object with the same outline. It doesn't work well for cases, where you want to inpaint an object, which didn't exist before. The reason is, that also some background will be inpainted and that looks different from the background on the original image due to the existing problem. For that reason you will see the inpainted part, when your suggested code is used.

@pesser It would be great to hear your opinion about that problem and even better, if you would have a suggestion how this could be fixed :).

@patil-suraj Would like to hear your opinion too. I already talked to @pcuenca over on discord. He told me, that you might have an idea how to fix the problem.

Thanks!

from stable-diffusion.

I noticed this too, what site are you referring to?

Probably compared to their tool at https://app.runwayml.com

Are you saying that Colab does not show distortion but the HuggingFace space does?

Colab and HuggingFace both return the distorted image. You can test this out by using a very low step count like 2 and you will see how the entire image is modified, not just the masked area.

I solved this problem mostly by reapplying the mask on the generated inpainted image:

final_inpainted = Image.composite(inpainted, image, mask)

from stable-diffusion.

I noticed this too, what site are you referring to? Are you saying that Colab does not show distortion but the HuggingFace space does?

from stable-diffusion.

Same issue here. I have a custom model that works perfectly on Automatic111 but not when I use it with diffusers.

Even if a use a mask, it still modifies all the faces and other small details all over the image.

from stable-diffusion.