biegert / comfyui-clipseg Goto Github PK

View Code? Open in Web Editor NEW

177.0 5.0 28.0 930 KB

ComfyUI CLIPSeg

License: Apache License 2.0

Python 100.00%

comfyui-clipseg's Introduction

Custom Nodes for ComfyUI: CLIPSeg and CombineSegMasks

This repository contains two custom nodes for ComfyUI that utilize the CLIPSeg model to generate masks for image inpainting tasks based on text prompts.

1. CLIPSeg

The CLIPSeg node generates a binary mask for a given input image and text prompt.

Inputs:

image: A torch.Tensor representing the input image.
text: A string representing the text prompt.
blur: A float value to control the amount of Gaussian blur applied to the mask.
threshold: A float value to control the threshold for creating the binary mask.
dilation_factor: A float value to control the dilation of the binary mask.

Outputs:

tensor_bw: A torch.Tensor representing the binary mask.
image_out_hm: A torch.Tensor representing the heatmap overlay on the input image.
image_out_bw: A torch.Tensor representing the binary mask overlay on the input image.

2. CombineSegMasks

The CombineSegMasks node combines two or optionally three masks into a single mask to improve masking of different areas.

Inputs:

image: A torch.Tensor representing the input image.
mask1: A torch.Tensor representing the first mask.
mask2: A torch.Tensor representing the second mask.
mask3 (optional): A torch.Tensor representing the third mask. Defaults to None.

Outputs:

combined_mask: A torch.Tensor representing the combined mask.
image_out_hm: A torch.Tensor representing the heatmap overlay of the combined mask on the input image.
image_out_bw: A torch.Tensor representing the binary mask overlay of the combined mask on the input image.

Installation

To use these custom nodes in your ComfyUI project, follow these steps:

Clone this repository or download the source code.
Put the clipseg.py file into your custom_nodes directory
???
Profit

Usage

Below is an example for the intended workflow. The json file for the example can be found inside the 'workflow' directory

Requirements

PyTorch
CLIPSeg
OpenCV
numpy
matplotlib

Make sure that you have the required libraries installed to the venv of ComfyUI.

comfyui-clipseg's People

Contributors

Stargazers

Watchers

comfyui-clipseg's Issues

Cannot import, No such file or directory

Cannot import C:...\ComfyUI\custom_nodes\ComfyUI-CLIPSeg module for custom nodes: [Errno 2] No such file or directory: 'C:\...\SD-ComfyUI\ComfyUI\custom_nodes\ComfyUI-CLIPSeg\init.py'

Cannot import init.py'

  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\Stable_difussion\\ComfyUI_windows_portable_nvidia_cu118_or_cpu\\ComfyUI_windows_portable\\ComfyUI\\custom_nodes\\ComfyUI-CLIPSeg\\__init__.py'

Cannot import D:\Stable_difussion\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CLIPSeg module for custom nodes: [Errno 2] No such file or directory: 'D:\\Stable_difussion\\ComfyUI_windows_portable_nvidia_cu118_or_cpu\\ComfyUI_windows_portable\\ComfyUI\\custom_nodes\\ComfyUI-CLIPSeg\\__init__.py'

Error occurred when executing CLIPSeg:

Failed to import transformers.models.vit.image_processing_vit because of the following error (look up to see its traceback):
A Message class can only inherit from Message

File "/content/drive/MyDrive/ComfyUI/execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/content/drive/MyDrive/ComfyUI/execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/content/drive/MyDrive/ComfyUI/execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/content/drive/MyDrive/ComfyUI/custom_nodes/clipseg.py", line 120, in segment_image
processor = CLIPSegProcessor.from_pretrained("CIDAS/clipseg-rd64-refined")
File "/usr/local/lib/python3.10/dist-packages/transformers/processing_utils.py", line 226, in from_pretrained
args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/processing_utils.py", line 268, in _get_arguments_from_pretrained
attribute_class = getattr(transformers_module, class_name)
File "/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py", line 1273, in getattr
value = getattr(module, name)
File "/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py", line 1272, in getattr
module = self._get_module(self._class_to_module[name])
File "/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py", line 1284, in _get_module
raise RuntimeError(

cv2.error: OpenCV(4.9.0) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\resize.cpp:3789: error: (-215:Assertion failed) !dsize.empty() in function 'cv::hal::resize'

!!! Exception during processing !!!
Traceback (most recent call last):
File "D:\Blender_ComfyUI\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Blender_ComfyUI\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Blender_ComfyUI\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Blender_ComfyUI\ComfyUI\custom_nodes\clipseg.py", line 154, in segment_image
heatmap_resized = resize_image(heatmap, dimensions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Blender_ComfyUI\ComfyUI\custom_nodes\clipseg.py", line 44, in resize_image
return cv2.resize(image, dimensions, interpolation=cv2.INTER_LINEAR)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

can not download model in china

Loading 1 new model
100%|██████████| 24/24 [00:14<00:00, 1.63it/s]
pytorch_model.bin: 2%|▏ | 12.6M/603M [04:18<3:22:01, 48.7kB/s]
!!! Exception during processing !!!
Traceback (most recent call last):
File "H:\ComfyUI\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "H:\ComfyUI\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "H:\ComfyUI\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "H:\ComfyUI\ComfyUI\custom_nodes\clipseg.py", line 121, in segment_image
model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined")
File "H:\ComfyUI\python_embeded\lib\site-packages\transformers\modeling_utils.py", line 2222, in from_pretrained
resolved_archive_file = cached_file(
File "H:\ComfyUI\python_embeded\lib\site-packages\transformers\utils\hub.py", line 409, in cached_file
resolved_file = hf_hub_download(
File "H:\ComfyUI\python_embeded\lib\site-packages\huggingface_hub\utils_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "H:\ComfyUI\python_embeded\lib\site-packages\huggingface_hub\file_download.py", line 1457, in hf_hub_download
http_get(
File "H:\ComfyUI\python_embeded\lib\site-packages\huggingface_hub\file_download.py", line 552, in http_get
raise EnvironmentError(
OSError: Consistency check failed: file should be of size 603143713 but has size 12608704 (pytorch_model.bin).
We are sorry for the inconvenience. Please retry download and pass force_download=True, resume_download=False as argument.
If the issue persists, please let us know by opening an issue on https://github.com/huggingface/huggingface_hub.

i found the model in huggingface

https://huggingface.co/CIDAS/clipseg-rd64-refined/tree/main

but which directory should i put it in

Stop update anymore?

Hi....this can't run in the latest Comfyui.
Could you update it ?
Thank you so much!

Anyway to exclude sth?

like: head
exclude: ear

`ReadTimeoutError` when using `FaceDetailer` + `CLIPSeg`

I'm reposting the issue from here:
ltdrdata/ComfyUI-Impact-Pack#116

I'm using Impact pack.
Very often (but not always), when I use FaceDetailer node with CLIPSegDetectorProvider - one of them spams ReadTimeoutError errors into console, with text like this:

Loads SAM model: E:\SD\ComfyUI-portable\ComfyUI\models\sams\sam_vit_b_01ec64.pth (device:Prefer GPU)
'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 621b36cd-5b11-4361-bbd3-c1f008d1a9e3)')' thrown while requesting HEAD https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/main/preprocessor_config.json
'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 50506107-44dc-4bf4-9fb7-48cf7ad9c640)')' thrown while requesting HEAD https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/main/tokenizer_config.json
'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: c5467a5f-4ec8-4074-a2b4-00765779fe5e)')' thrown while requesting HEAD https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/main/config.json
# of Detected SEGS: 1
Detailer: segment upscale for ((419, 390)) | crop region (838, 780) x 1.8331748251748252 -> (1536, 1429)

The nodes author claims that they are just wrappers on top of CLIPSeg and the actual issue is here.

The error in question is not from FaceDetailer but rather from CLIPSeg. CLIPSeg has a part where it downloads the model through the network initially, so if there is no internet connection, this can become an issue.

I can't reproduce the issue on demand. It either happens or not, with no obvious cause. Sometimes my whole graph (heavily utilizing FaceDetailer + CLIPSeg) just works like a breeze. Sometimes, when I have a long sequence of such nodes, each of them spams the same set of errors. My best guess is I'm getting antispam-banned at huggingface after a while.

And looks like, the only thing CLIPSeg tries to retrieve is a couple of json files. Why don't just download them once?

Whether there is an incompatibility problem with ip-adapter plus

Raise error: (-215:Assertion failed) !dsize.empty() in function 'cv::hal::resize'

ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "E:\DEV\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "E:\DEV\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "E:\DEV\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "E:\DEV\ComfyUI_windows_portable\ComfyUI\custom_nodes\clipsegnode.py", line 216, in combine_masks
heatmap_resized = resize_image(heatmap, dimensions)
File "E:\DEV\ComfyUI_windows_portable\ComfyUI\custom_nodes\clipsegnode.py", line 44, in resize_image
return cv2.resize(image, dimensions, interpolation=cv2.INTER_LINEAR)
cv2.error: OpenCV(4.8.1) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\resize.cpp:3699: error: (-215:Assertion failed) !dsize.empty() in function 'cv::hal::resize'

Regarding OpenCV3699, cv::hal::resize

how to solve this problem？？？？？

Import Failed on Comfyui-Clipseg with following error

File "/Users/fredlefevre/AI/ComfyUI/custom_nodes/ComfyUI-CLIPSeg/init.py", line 16, in
from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available
ImportError: attempted relative import beyond top-level package

Cannot import /Users/fredlefevre/AI/ComfyUI/custom_nodes/ComfyUI-CLIPSeg module for custom nodes: attempted relative import beyond top-level package

Thanks for any help

Exception during processing!!! HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /CIDAS/clipseg-rd64-refined/resolve/main/model.safetensors

Exception during processing!!! HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /CIDAS/clipseg-rd64-refined/resolve/main/model.safetensors (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x000001F558555CC0>, 'Connection to huggingface.co timed out. (connect timeout=10)'))

Mosaics appear when cutting out images

Have you ever encountered this kind of problem? This kind of mosaic result occurs when cutting out pictures.

Create Masks like ComfyUI

You should be creating your masks like ComfyUI creates them, so they're compatible with nodes that process them.

https://github.com/comfyanonymous/ComfyUI/blob/master/nodes.py#L990

They are inverted, for one.

(IMPORT FAILED): D:\ComfyUI\custom_nodes\ComfyUI-CLIPSeg

(venv) D:\ComfyUI\custom_nodes\ComfyUI-CLIPSeg>pip install -r requirements.txt
Looking in indexes: https://mirrors.cloud.tencent.com/pypi/simple
Requirement already satisfied: matplotlib==3.7.1 in d:\comfyui\venv\lib\site-packages (from -r requirements.txt (line 2)) (3.7.1)
Requirement already satisfied: matplotlib-inline==0.1.6 in d:\comfyui\venv\lib\site-packages (from -r requirements.txt (line 3)) (0.1.6)
Collecting numpy==1.24.2 (from -r requirements.txt (line 4))
Using cached https://mirrors.cloud.tencent.com/pypi/packages/fa/df/53e8c0c8ccecf360b827a3d2b1b6060644c635c3149a9d6415a6fe4ccf44/numpy-1.24.2-cp310-cp310-win_amd64.whl (14.8 MB)
Requirement already satisfied: open-clip-torch==2.16.0 in d:\comfyui\venv\lib\site-packages (from -r requirements.txt (line 5)) (2.16.0)
Collecting opencv-python==4.7.0.72 (from -r requirements.txt (line 6))
Using cached https://mirrors.cloud.tencent.com/pypi/packages/36/98/fab8d982e2e2b57bdebcad64c7e5b5a14ac91c657cac509b9cf3fbea49d2/opencv_python-4.7.0.72-cp37-abi3-win_amd64.whl (38.2 MB)
Collecting Pillow==9.4.0 (from -r requirements.txt (line 7))
Using cached https://mirrors.cloud.tencent.com/pypi/packages/5e/7c/293136a5171800001be33c21a51daaca68fae954b543e2c015a6bb81a716/Pillow-9.4.0-cp310-cp310-win_amd64.whl (2.5 MB)
Requirement already satisfied: pytorch-lightning==2.0.0 in d:\comfyui\venv\lib\site-packages (from -r requirements.txt (line 8)) (2.0.0)
ERROR: Could not find a version that satisfies the requirement torch==2.2.0+cu121 (from versions: 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1)
ERROR: No matching distribution found for torch==2.2.0+cu121

Workflow for video images?

I just managed to get Clipseg to work for still images, and it’s such a neat tool! I’m trying to set up a workflow to generate masks for video clips too.

First of all, I’m completely new to this so apologize if this is a plain silly idea for this particular custom node, but how would one go about to load a video instead of just a single image through the clipseg node?

When trying this, I get an error saying the "input array is receiving more dimensions than expected”:

Error occurred when executing CLIPSeg:

Too many dimensions: 4 > 3.

File "E:\COMFYUI\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\COMFYUI\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\COMFYUI\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\COMFYUI\ComfyUI\custom_nodes\clipseg.py", line 118, in segment_image
i = Image.fromarray(image_np, mode="RGB")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\snurrebassen\AppData\Local\Programs\Python\Python312\Lib\site-packages\PIL\Image.py", line 3103, in fromarray
raise ValueError(msg)

I was wondering if anyone has suggestions or workarounds for this issue.

Error Updating nodes

tried downloading and installing latest CLIPSeg nodes and it fails on Comfyui startup with error Cannot import C:\AI\ComfyUI\ComfyUI\custom_nodes\ComfyUI-CLIPSeg-main module for custom nodes: [Errno 2] No such file or directory: 'C:\AI\ComfyUI\ComfyUI\custom_nodes\ComfyUI-CLIPSeg-main\init.py'.

How do I resolve this error?
thanks