Small experiment on combining CLIP with SAM to do open-vocabulary image segmentation.

• Segment Anything Model (SAM)
• Contrastive Language-Image Pre-Training (CLIP)

The approach is to first identify all the parts of an image using SAM, and then use CLIP to find the ones that best match a specific description.

1. Download weights and place them in this repos root.

2. Install dependencies:

    pip install torch opencv-python Pillow
    pip install git+https://github.com/openai/CLIP.git
    pip install git+https://github.com/facebookresearch/segment-anything.git

3. Run Notebook main.ipynb

Example output for prompt "kiwi"

Example Image Source