This extension aim for helping stable diffusion webui users to use segment anything to do stable diffusion inpainting.

• 2023/04/12: [Feature] Mask expansion enabled. Thanks @jordan-barrett-jm for your great contribution!
• 2023/04/12: [Feature] Experimental GroundingDINO support released in GroundingDINO branch. The current implementation is VERY unstable. I recommend waiting until it is merged into master branch to use it.

Thanks for suggestions from GitHub Issues, reddit and bilibili to make this extension better.

• Support mask expansion
• Support text->detection->segmentation from Grounded Segment Anything and merge to master branch
• Support API as mentioned in #15
• Support WebUI older commits (e.g. a9fed7c364061ae6efb37f797b6b522cb3cf7aa2)
• Connect to ControlNet
• Batch process to support LoRA/LyCORIS training

If you are unable to add dot, observe list index out of range error on your terminal, or any other error, the most probable reason is that your WebUI is outdated (such as you are using this commitment: a9fed7c364061ae6efb37f797b6b522cb3cf7aa2).

In most cases, updating your WebUI can solve your problem. Before you submit your issue and before I release support for some old version of WebUI, I ask that you firstly check your version of your WebUI.

Download this extension to ${sd-webui}/extensions use whatever way you like (git clone or install from UI)

Download segment-anything model from here to ${sd-webui-segment-anything}/models/sam. Do not change model name, otherwise this extension may fail due to a bug inside segment anything.

To give you a reference, vit_h is 2.56GB, vit_l is 1.25GB, vit_b is 375MB. I myself tested vit_h on NVIDIA 3090 Ti which is good. If you encounter VRAM problem, you should switch to smaller models.

• Launch webui and switch to img2img mode.
• Upload your image and add prompts on the image (MUST). Left click for positive prompt (black dot), right click for negative prompt (red dot), left click any dot again to cancel the prompt. If you forgot to add prompts, there will be exceptions on your terminal.
• Click Preview Segmentation button
• Choose your favorite segmentation and check Copy to Inpaint Upload
• Optionally check Expand Mask and specify the amount, then click Update Mask
• Switch to Inpaint upload. There is no need to upload another image or mask, just leave them blank. Write your prompt, configurate and click Generate.

Disclaimer: I have not thoroughly tested this extension, so there might be bugs. Bear with me while I'm fixing them :)

If you encounter a bug, please submit a issue. Please at least provide your WebUI version, your extension version, your browser version, errors on your browser console log if there is any, error on your terminal log if there is any, to make sure that I can find a solution faster.

I welcome any contribution. Please submit a pull request if you want to contribute