hangover3832 / comfyui-hangover-moondream Goto Github PK

View Code? Open in Web Editor NEW

34.0 4.0 2.0 2 MB

Moondream is a lightweight multimodal large language model

Home Page: https://github.com/Hangover3832/ComfyUI-Hangover-Moondream

License: Apache License 2.0

Python 100.00%

comfyui moondream multimodal-large-language-models stable-diffusion

comfyui-hangover-moondream's Introduction

ConfyUI-Hangover-Moondream

Moondream is a lightweight multimodal large languge model.

⚠️ WARNING: Additional python code will be downloaded from huggingface and executed. You have to trust this creator if you want to use this node!

👍 For testing, research and fun.

Updates

Now passing proper prompt to the model 🐞
Model/code update moondream2. This should resolve issues with the 'Tensor size mismatch' error when using newer versions of transformers.Make sure that you select the moondream2 model within the node to receive the updated files from huggingface. If moondream2 works fine, you can remove the old model files, usually located in the user\.cache\huggingface folder: hub\models--vikhyatk--moondream1 and modules\transformers_modules\vikhyatk\moondream1 to save disk space.🔄
End support for moondream1 in order to change the license for this repository to Apache 2.0.
New multi prompt feature: Ask several questions at once. Just place each prompt on a new line. 🔄
Added model revision selector. 🔄
Added model temperature selector. Set this value to 0 to use the models default temperature. 🔄

To do

comfyui-hangover-moondream's People

Contributors

Stargazers

Watchers

Forkers

toxicwind haohaocreates

comfyui-hangover-moondream's Issues

Last UPDATE comfyui 20.03.24

CPU

Error occurred when executing Moondream:

The size of tensor a (750) must match the size of tensor b (751) at non-singleton dimension 1

File "E:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-0246\utils.py", line 381, in new_func
res_value = old_func(*final_args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 65, in map_node_over_list
results.append(getattr(obj, func)(**input_data_all))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py", line 121, in run
res=self.moondream.answer_question(image_embeds, question,self.tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\moondream.py", line 93, in answer_question
answer = self.generate(
^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\moondream.py", line 77, in generate
output_ids = self.text_model.generate(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\generation\utils.py", line 1544, in generate
device=inputs_tensor.device,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\generation\utils.py", line 2404, in greedy_search
if this_peer_finished and not synced_gpus:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 709, in forward
hidden_states = self.transformer(
^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 675, in forward
else func(*args)
^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 541, in forward
attn_outputs = self.mixer(
^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 514, in forward
attn_output_function(x, past_key_values, attention_mask)
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 494, in _forward_cross_attn
return attn_func(
^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 491, in
else lambda fn, *args, **kwargs: fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 318, in forward
padding_mask.masked_fill_(key_padding_mask, 0.0)

CUDA

Error occurred when executing Moondream:

The expanded size of the tensor (750) must match the existing size (751) at non-singleton dimension 1. Target sizes: [1, 750]. Tensor sizes: [1, 751]

File "E:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-0246\utils.py", line 381, in new_func
res_value = old_func(*final_args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 65, in map_node_over_list
results.append(getattr(obj, func)(**input_data_all))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py", line 121, in run
res=self.moondream.answer_question(image_embeds, question,self.tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\moondream.py", line 93, in answer_question
answer = self.generate(
^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\moondream.py", line 77, in generate
output_ids = self.text_model.generate(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\generation\utils.py", line 1544, in generate
device=inputs_tensor.device,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\generation\utils.py", line 2404, in greedy_search
if this_peer_finished and not synced_gpus:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 709, in forward
hidden_states = self.transformer(
^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 675, in forward
else func(*args)
^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 541, in forward
attn_outputs = self.mixer(
^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 514, in forward
attn_output_function(x, past_key_values, attention_mask)
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 494, in _forward_cross_attn
return attn_func(
^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 491, in
else lambda fn, *args, **kwargs: fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "E:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-moondream\nodes\MoondreamNode.py\../..\moondream\modeling_phi.py", line 318, in forward
padding_mask.masked_fill_(key_padding_mask, 0.0)

couldn't find it in the cached files and it looks like vikhyatk/moondream2 is not the path to a directory containing a file named config.json.

Describe the bug
Using the given directory but no model found：

Desktop (please complete the following information):

OS: windows

Python packages:

You have to trust remote code execution to use this node!

How do I do that in comfy?

Tensor size mismatch

Errors out with a mismatch in the size of the tensor. It happens both on CPU and GPU. Here's the terminal output:

ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "ComfyUI/execution.py", line 152, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "ComfyUI/execution.py", line 82, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "ComfyUI/execution.py", line 75, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "ComfyUI/custom_nodes/ComfyUI-Hangover-Moondream/ho_moondream.py", line 71, in interrogate
    answer = self.model.answer_question(enc_image, prompt, self.tokenizer)
  File "/home/ricperry1/.cache/huggingface/modules/transformers_modules/vikhyatk/moondream1/f6e9da68e8f1b78b8f3ee10905d56826db7a5802/moondream.py", line 93, in answer_question
    answer = self.generate(
  File "/home/ricperry1/.cache/huggingface/modules/transformers_modules/vikhyatk/moondream1/f6e9da68e8f1b78b8f3ee10905d56826db7a5802/moondream.py", line 77, in generate
    output_ids = self.text_model.generate(
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "ComfyUI/venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 1544, in generate
    return self.greedy_search(
  File "ComfyUI/venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 2404, in greedy_search
    outputs = self(
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1529, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ricperry1/.cache/huggingface/modules/transformers_modules/vikhyatk/moondream1/f6e9da68e8f1b78b8f3ee10905d56826db7a5802/modeling_phi.py", line 709, in forward
    hidden_states = self.transformer(
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1529, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ricperry1/.cache/huggingface/modules/transformers_modules/vikhyatk/moondream1/f6e9da68e8f1b78b8f3ee10905d56826db7a5802/modeling_phi.py", line 675, in forward
    else func(*args)
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1529, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ricperry1/.cache/huggingface/modules/transformers_modules/vikhyatk/moondream1/f6e9da68e8f1b78b8f3ee10905d56826db7a5802/modeling_phi.py", line 541, in forward
    attn_outputs = self.mixer(
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1529, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ricperry1/.cache/huggingface/modules/transformers_modules/vikhyatk/moondream1/f6e9da68e8f1b78b8f3ee10905d56826db7a5802/modeling_phi.py", line 514, in forward
    attn_output_function(x, past_key_values, attention_mask)
  File "/home/ricperry1/.cache/huggingface/modules/transformers_modules/vikhyatk/moondream1/f6e9da68e8f1b78b8f3ee10905d56826db7a5802/modeling_phi.py", line 494, in _forward_cross_attn
    return attn_func(
  File "/home/ricperry1/.cache/huggingface/modules/transformers_modules/vikhyatk/moondream1/f6e9da68e8f1b78b8f3ee10905d56826db7a5802/modeling_phi.py", line 491, in <lambda>
    else lambda fn, *args, **kwargs: fn(*args, **kwargs)
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1529, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    return forward_call(*args, **kwargs)
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "ComfyUI/venv/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/ricperry1/.cache/huggingface/modules/transformers_modules/vikhyatk/moondream1/f6e9da68e8f1b78b8f3ee10905d56826db7a5802/modeling_phi.py", line 318, in forward
    padding_mask.masked_fill_(key_padding_mask, 0.0)
RuntimeError: The expanded size of the tensor (760) must match the existing size (761) at non-singleton dimension 1.  Target sizes: [1, 760].  Tensor sizes: [1, 761]

Error on first use

trust_remote_code is set to true. And yes, the error messages ends like that. Seems incomplete. This is the error message shown in ComfyUI.

Error occurred when executing Moondream Interrogator (NO COMMERCIAL USE):

cannot import name 'ToImage' from 'torchvision.transforms.v2' (F:\AI\ComfyUI\python_embeded\lib\site-packages\torchvision\transforms\v2_init_.py)

File "F:\AI\ComfyUI\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "F:\AI\ComfyUI\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "F:\AI\ComfyUI\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "F:\AI\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Hangover-Moondream\ho_moondream.py", line 56, in interrogate
self.model = AutoModelForCausalLM.from_pretrained(huggingface_model, trust_remote_code=trust_remote_code).to(dev)
File "F:\AI\ComfyUI\python_embeded\lib\site-packages\transformers\models\auto\auto_factory.py", line 455, in from_pretrained
model_class = get_class_from_dynamic_module(
File "F:\AI\ComfyUI\python_embeded\lib\site-packages\transformers\dynamic_module_utils.py", line 374, in get_class_from_dynamic_module
return get_class_in_module(class_name, final_module.replace(".py", ""))
File "F:\AI\ComfyUI\python_embeded\lib\site-packages\transformers\dynamic_module_utils.py", line 147, in get_class_in_module
module = importlib.import_module(module_path)
File "importlib_init_.py", line 126, in import_module
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "C:\Users\amari/.cache\huggingface\modules\transformers_modules\vikhyatk\moondream1\f6e9da68e8f1b78b8f3ee10905d56826db7a5802\moondream.py", line 3, in
from .vision_encoder import VisionEncoder
File "C:\Users\amari/.cache\huggingface\modules\transformers_modules\vikhyatk\moondream1\f6e9da68e8f1b78b8f3ee10905d56826db7a5802\vision_encoder.py", line 5, in
from torchvision.transforms.v2 import (

Question: pre-existing copy of the model.

Not a bug, but a question. If I already have a local copy of the moondream model, what do i need to change to make it point to that instead of the cached version that it tries to download?

Moondream Interrogator dont start

Hi, @Hangover3832 yesterday I install your extensión but when I start queue the moondream 2 loading it dont do anything, can you help me? (Moondream interrogator note have trust remote access “TRUE”

got prompt
[rgthree] Using rgthree's optimized recursive execution.
[rgthree] First run patching recursive_output_delete_if_changed and recursive_will_execute.
[rgthree] Note: If execution seems broken due to forward ComfyUI changes, you can disable the optimization from rgthree settings in ComfyUI.
C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\functional.py:507: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ..\aten\src\ATen\native\TensorShape.cpp:3550.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
moondream: loading model vikhyatk/moondream2, please stand by....
!!! Exception during processing !!!
Traceback (most recent call last):
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\urllib3\connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
raise err
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\urllib3\connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\urllib3\connectionpool.py", line 404, in _make_request
self._validate_conn(conn)
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\urllib3\connectionpool.py", line 1058, in _validate_conn
conn.connect()
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\urllib3\connection.py", line 363, in connect
self.sock = conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\urllib3\connection.py", line 179, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x00000241ABC77910>, 'Connection to huggingface.co timed out. (connect timeout=10)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\requests\adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\urllib3\connectionpool.py", line 799, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\urllib3\util\retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /vikhyatk/moondream2/resolve/main/config.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x00000241ABC77910>, 'Connection to huggingface.co timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\huggingface_hub\file_download.py", line 1238, in hf_hub_download
metadata = get_hf_file_metadata(
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\huggingface_hub\utils_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\huggingface_hub\file_download.py", line 1631, in get_hf_file_metadata
r = _request_wrapper(
^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\huggingface_hub\file_download.py", line 385, in _request_wrapper
response = _request_wrapper(
^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\huggingface_hub\file_download.py", line 408, in _request_wrapper
response = get_session().request(method=method, url=url, **params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\requests\sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\requests\sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\huggingface_hub\utils_http.py", line 67, in send
return super().send(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\requests\adapters.py", line 507, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /vikhyatk/moondream2/resolve/main/config.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x00000241ABC77910>, 'Connection to huggingface.co timed out. (connect timeout=10)'))"), '(Request ID: 33bd3199-2032-4a12-abdd-16996d5c9242)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\utils\hub.py", line 398, in cached_file
resolved_file = hf_hub_download(
^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\huggingface_hub\utils_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\huggingface_hub\file_download.py", line 1371, in hf_hub_download
raise LocalEntryNotFoundError(
huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Hangover-Moondream\ho_moondream.py", line 56, in interrogate
self.model = AutoModelForCausalLM.from_pretrained(huggingface_model, trust_remote_code=trust_remote_code).to(dev)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 521, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 1111, in from_pretrained
config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\configuration_utils.py", line 633, in get_config_dict
config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\configuration_utils.py", line 688, in _get_config_dict
resolved_config_file = cached_file(
^^^^^^^^^^^^
File "C:\Users\joela\OneDrive\Documentos\Stable Diffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\utils\hub.py", line 441, in cached_file
raise EnvironmentError(
OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like vikhyatk/moondream2 is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.

Prompt executed in 361.53 seconds

Object of type PhiConfig is not JSON serializable

I get the following error when running Moondream Interrogator:

Error occurred when executing Moondream Interrogator:

Object of type PhiConfig is not JSON serializable

File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\apps\SD-WebUI\ComfyUI\custom_nodes\ComfyUI-Hangover-Moondream\ho_moondream.py", line 108, in interrogate
self.model = AutoModel.from_pretrained(
File "D:\apps\Python\Python310\lib\site-packages\transformers\models\auto\auto_factory.py", line 434, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
File "D:\apps\Python\Python310\lib\site-packages\transformers\models\auto\configuration_auto.py", line 871, in from_pretrained
return config_class.from_pretrained(pretrained_model_name_or_path, **kwargs)
File "D:\apps\Python\Python310\lib\site-packages\transformers\configuration_utils.py", line 545, in from_pretrained
return cls.from_dict(config_dict, **kwargs)
File "D:\apps\Python\Python310\lib\site-packages\transformers\configuration_utils.py", line 712, in from_dict
logger.info(f"Model config {config}")
File "D:\apps\Python\Python310\lib\site-packages\transformers\configuration_utils.py", line 744, in __repr__
return f"{self.__class__.__name__} {self.to_json_string()}"
File "D:\apps\Python\Python310\lib\site-packages\transformers\configuration_utils.py", line 816, in to_json_string
return json.dumps(config_dict, indent=2, sort_keys=True) + "\n"
File "D:\apps\Python\Python310\lib\json\__init__.py", line 238, in dumps
**kw).encode(obj)
File "D:\apps\Python\Python310\lib\json\encoder.py", line 201, in encode
chunks = list(chunks)
File "D:\apps\Python\Python310\lib\json\encoder.py", line 431, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "D:\apps\Python\Python310\lib\json\encoder.py", line 405, in _iterencode_dict
yield from chunks
File "D:\apps\Python\Python310\lib\json\encoder.py", line 438, in _iterencode
o = _default(o)
File "D:\apps\Python\Python310\lib\json\encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '

Unknown model error!

How to install the updated model and overcome this issues?

can you please provide the path where to place the model?

Thanks

Cannot import name `ToImage` error

I got the following error when executing your node within a workflow. Comfy is updated to the latest, and have the following parameters.

Error occurred when executing Moondream Interrogator:

cannot import name 'ToImage' from 'torchvision.transforms.v2' (C:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torchvision\transforms\v2\__init__.py)

File "C:\sd\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\sd\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\sd\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\sd\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Hangover-Moondream\ho_moondream.py", line 56, in interrogate
self.model = AutoModelForCausalLM.from_pretrained(huggingface_model, trust_remote_code=trust_remote_code).to(dev)
File "C:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\models\auto\auto_factory.py", line 548, in from_pretrained
model_class = get_class_from_dynamic_module(
File "C:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\dynamic_module_utils.py", line 500, in get_class_from_dynamic_module
return get_class_in_module(class_name, final_module.replace(".py", ""))
File "C:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\dynamic_module_utils.py", line 200, in get_class_in_module
module = importlib.import_module(module_path)
File "importlib\__init__.py", line 126, in import_module
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "C:\Users\dali\.cache\huggingface\modules\transformers_modules\vikhyatk\moondream2\0691a810b729384352a1bb0db09a43bb2d80e283\moondream.py", line 2, in
from .vision_encoder import VisionEncoder
File "C:\Users\dali\.cache\huggingface\modules\transformers_modules\vikhyatk\moondream2\0691a810b729384352a1bb0db09a43bb2d80e283\vision_encoder.py", line 5, in
from torchvision.transforms.v2 import (

You have to trust remote code to use this node!

Error occurred when executing Moondream Interrogator:

You have to trust remote code to use this node!

File "D:\Program Files\ComfyUI_base\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Program Files\ComfyUI_base\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Program Files\ComfyUI_base\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Program Files\ComfyUI_base\ComfyUI\custom_nodes\ComfyUI-Hangover-Moondream\ho_moondream.py", line 74, in interrogate
raise ValueError("You have to trust remote code to use this node!")
Seeking advice, encountering this problem

hangover3832 / comfyui-hangover-moondream Goto Github PK

comfyui-hangover-moondream's Introduction

ConfyUI-Hangover-Moondream

Updates

To do

comfyui-hangover-moondream's People

Contributors

Stargazers

Watchers

Forkers

comfyui-hangover-moondream's Issues

Recommend Projects

Recommend Topics

Recommend Org