cubiq / comfyui_ipadapter_plus Goto Github PK
View Code? Open in Web Editor NEWLicense: GNU General Public License v3.0
comfyui_ipadapter_plus's People
Contributors
Stargazers
Watchers
Forkers
nomand jags111 guyuehu2304 storyicon melmass clic-ethiopia draw-your-dream kosinkadink aaronpanxiaofeng ka-de arron17 entmike kodavn jaideepjoshi m1kep crystalwizard xingren23 camenduru xeeroxxx hito0512 ttulttul niceboy19 xymfei flowtyone iminetd37 marjelll eltociear agentsolidnonfutou duyyguu prestongoren mikigit11 kalasforever xiaohu2015 beeag80 championderp73 udogger bornedbalarec janssma75 roseswasabiquanttrendy syntherperfectiveq limetaryd tutumomo hhy5277 slickytail enardleyno bodhihu sm-da pikachudratini nouira21 lvxing75 ciihom hjsybyq shadowcz007 jiaxi-dai bencoster capsadmin barleyj21 coffeeyou ameasure hippopenny kellhuang mindfulpol rvion 8funtik8 soliver84 zigzag-tech ro728 yiming992 codecyh nidefawl lililibiu peruwski erikluo differlong isurajboss harduex sagiodev unique15 rwubarhblp516 thinkdiffusion vxinhao faiseh00d wfjsw tymczas assassindesign brunokk smirnoff comfyorg zhuangwuxin ailty drtekiti wetetghfd aravind-h-v thd-technologies aegis72 barrtender vinroy89 erendrgnl i3doit stastiemcomfyui_ipadapter_plus's Issues
[Windows] Runs using the same seed produce different results with certain negative embeddings
It seems certain negative embeddings cause repeated runs, using the same configuration, to give different images as result. A friend checked this on a linux machine and it seemed to work fine there.
I've tried changing the denoise settings or other settings but the culprit seems to be in negative embeddings.
I've attached a small workflow which runs 2 samplers, which should, with the same configuration, return the same image.
but instead the following images are returned.
1.
2.
They're identical, but slightly different.
I used the following pic as input.
If i replace the negative embedding embedding:ng_deepnegative_v1_75t with embedding:EasyNegative, it does seem to work. So it seems very specific.
Do you have any idea what could cause this quite particular behavior?
Do you think this is a better face swapper vs ReActor Node for ComfyUI
I'm not sure which one to use? Do you have any insights into pros vs cons?
Feature Req: Add randomization to embed selection loader node
I have a directory of embeds and want to have the system select a random one for each generation. This would be amazing!
IP adapter plus model batch images Seems to have no effect
and ip-adapter-plus-face_sd15 model also like this
Error with the new SDXL plus model
Traceback (most recent call last):
File "/home/matteo/bin/ComfyUI/execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/matteo/bin/ComfyUI/execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/matteo/bin/ComfyUI/execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/matteo/bin/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 330, in apply_ipadapter
self.ipadapter = IPA(
^^^^
File "/home/matteo/bin/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 179, in __init__
self.image_proj_model.load_state_dict(ipadapter_model["image_proj"])
File "/home/matteo/bin/ComfyUI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Resampler:
size mismatch for latents: copying a param with shape torch.Size([1, 16, 1280]) from checkpoint, the shape in current model is torch.Size([1, 16, 2048]).
size mismatch for proj_in.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 1280]).
size mismatch for proj_in.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for proj_out.weight: copying a param with shape torch.Size([2048, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 2048]).
size mismatch for layers.0.0.norm1.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.norm1.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.norm2.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.norm2.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.to_q.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([768, 2048]).
size mismatch for layers.0.0.to_kv.weight: copying a param with shape torch.Size([2560, 1280]) from checkpoint, the shape in current model is torch.Size([1536, 2048]).
size mismatch for layers.0.0.to_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 768]).
size mismatch for layers.0.1.0.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.1.0.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.1.1.weight: copying a param with shape torch.Size([5120, 1280]) from checkpoint, the shape in current model is torch.Size([8192, 2048]).
size mismatch for layers.0.1.3.weight: copying a param with shape torch.Size([1280, 5120]) from checkpoint, the shape in current model is torch.Size([2048, 8192]).
size mismatch for layers.1.0.norm1.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.0.norm1.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.0.norm2.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.0.norm2.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.0.to_q.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([768, 2048]).
size mismatch for layers.1.0.to_kv.weight: copying a param with shape torch.Size([2560, 1280]) from checkpoint, the shape in current model is torch.Size([1536, 2048]).
size mismatch for layers.1.0.to_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 768]).
size mismatch for layers.1.1.0.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.1.0.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.1.1.weight: copying a param with shape torch.Size([5120, 1280]) from checkpoint, the shape in current model is torch.Size([8192, 2048]).
size mismatch for layers.1.1.3.weight: copying a param with shape torch.Size([1280, 5120]) from checkpoint, the shape in current model is torch.Size([2048, 8192]).
size mismatch for layers.2.0.norm1.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.0.norm1.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.0.norm2.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.0.norm2.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.0.to_q.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([768, 2048]).
size mismatch for layers.2.0.to_kv.weight: copying a param with shape torch.Size([2560, 1280]) from checkpoint, the shape in current model is torch.Size([1536, 2048]).
size mismatch for layers.2.0.to_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 768]).
size mismatch for layers.2.1.0.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.1.0.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.1.1.weight: copying a param with shape torch.Size([5120, 1280]) from checkpoint, the shape in current model is torch.Size([8192, 2048]).
size mismatch for layers.2.1.3.weight: copying a param with shape torch.Size([1280, 5120]) from checkpoint, the shape in current model is torch.Size([2048, 8192]).
size mismatch for layers.3.0.norm1.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.0.norm1.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.0.norm2.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.0.norm2.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.0.to_q.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([768, 2048]).
size mismatch for layers.3.0.to_kv.weight: copying a param with shape torch.Size([2560, 1280]) from checkpoint, the shape in current model is torch.Size([1536, 2048]).
size mismatch for layers.3.0.to_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 768]).
size mismatch for layers.3.1.0.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.1.0.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.1.1.weight: copying a param with shape torch.Size([5120, 1280]) from checkpoint, the shape in current model is torch.Size([8192, 2048]).
size mismatch for layers.3.1.3.weight: copying a param with shape torch.Size([1280, 5120]) from checkpoint, the shape in current model is torch.Size([2048, 8192]).
IPAdapterApplyEncoded "SDXL+vit-h+sd1.5clip vision" error
If you don't use "Encode IPAdapter Image" and "Apply IPAdapter from Encoded", it works fine, but then you can't use img weights.
Of course, when using a CLIP Vision Encode node with a CLIP Vision model that uses SD1.5, and the basemodel is an SDXL model, there would have been an error and it wouldn't have run.
So regarding your image weighting feature, can it be implemented in this case?
Error occurred when executing IPAdapterApplyEncoded:
Error(s) in loading state_dict for Resampler:
size mismatch for proj_in.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1024]).
File "D:\StableSwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\StableSwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\StableSwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\StableSwarmUI\dlbackend\comfy\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 316, in apply_ipadapter
self.ipadapter = IPAdapter(
^^^^^^^^^^
File "D:\StableSwarmUI\dlbackend\comfy\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 161, in init
self.image_proj_model.load_state_dict(ipadapter_model["image_proj"])
File "D:\StableSwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 2152, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
Where take the other parameter files ? (Not clear from install instructions)
I installed ComfyUI in order to test this very promissing plugin (before I was using automatic 1111 webgui for stable diffusion, so I have no experience with ConfyUI)
When I load the workflow form json file there are set these models:
vae-ft-mse-840000-ema-pruned.safetensors
control_v11p_sd15_canny_fp16.safetensors
where can I download them?
Cannot import /home/ec2-user/demo/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus module for custom nodes: cannot import name 'clip_preprocess' from 'comfy.clip_vision' (/home/ec2-user/demo/ComfyUI/comfy/clip_vision.py)
After I installed the plug-in, the value could not be normal and an error message was reported :
Traceback (most recent call last):
File "/home/ec2-user/demo/ComfyUI/nodes.py", line 1735, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/home/ec2-user/demo/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/init.py", line 1, in
from .IPAdapterPlus import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
File "/home/ec2-user/demo/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 7, in
from comfy.clip_vision import clip_preprocess
ImportError: cannot import name 'clip_preprocess' from 'comfy.clip_vision' (/home/ec2-user/demo/ComfyUI/comfy/clip_vision.py)
Cannot import /home/ec2-user/demo/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus module for custom nodes: cannot import name 'clip_preprocess' from 'comfy.clip_vision' (/home/ec2-user/demo/ComfyUI/comfy/clip_vision.py)
KeyError: ((1, 1, 224, 3), '|u1')
When I execute IPAdapter_workflow.json,I got this error:
Traceback (most recent call last):
File "H:\AI\ComfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\PIL\Image.py", line 3080, in fromarray
mode, rawmode = _fromarray_typemap[typekey]
KeyError: ((1, 1, 224, 3), '|u1')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "H:\AI\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "H:\AI\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "H:\AI\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "H:\AI\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 262, in apply_ipadapter
clip_embed_zeroed = zeroed_hidden_states(clip_vision)
File "H:\AI\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 99, in zeroed_hidden_states
inputs = clip_vision.processor(images=image, return_tensors="pt")
File "H:\AI\ComfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\image_processing_utils.py", line 446, in call
return self.preprocess(images, **kwargs)
File "H:\AI\ComfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\models\clip\image_processing_clip.py", line 318, in preprocess
images = [self.resize(image=image, size=size, resample=resample) for image in images]
File "H:\AI\ComfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\models\clip\image_processing_clip.py", line 318, in
images = [self.resize(image=image, size=size, resample=resample) for image in images]
File "H:\AI\ComfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\models\clip\image_processing_clip.py", line 144, in resize
return resize(image, size=output_size, resample=resample, data_format=data_format, **kwargs)
File "H:\AI\ComfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\image_transforms.py", line 275, in resize
image = to_pil_image(image)
File "H:\AI\ComfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\image_transforms.py", line 163, in to_pil_image
return PIL.Image.fromarray(image)
File "H:\AI\ComfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\PIL\Image.py", line 3083, in fromarray
raise TypeError(msg) from e
TypeError: Cannot handle this data type: (1, 1, 224, 3), |u1
Can you tell me what went wrong?
refusing to merge unrelated histories
I've deleted the legacy IPAdapter plugin after reading that this plus version uses less memory, but I get this error when installing it:
C:\ComfyUI_windows_portable\ComfyUI\custom_nodes>git pull https://github.com/cubiq/ComfyUI_IPAdapter_plus
remote: Enumerating objects: 131, done.
remote: Counting objects: 100% (131/131), done.
remote: Compressing objects: 100% (87/87), done.
remote: Total 131 (delta 78), reused 92 (delta 41), pack-reused 0
Receiving objects: 100% (131/131), 1.91 MiB | 2.29 MiB/s, done.
Resolving deltas: 100% (78/78), done.
From https://github.com/cubiq/ComfyUI_IPAdapter_plus
* branch HEAD -> FETCH_HEAD
fatal: refusing to merge unrelated histories
'CLIPVisionModelOutput' object has no attribute 'penultimate_hidden_states'
Hi, thanks for the great project. It looks really cool.
When I try to run the inpaint example with sdxl, it always run to the error. I tried all ip-adapter model and clip vision model. always get this error.
Error occurred when executing IPAdapterApply:
'CLIPVisionModelOutput' object has no attribute 'penultimate_hidden_states'
File "/home/ubuntu/comfyUI2/ComfyUI/execution.py", line 151, in recursive_execute
File "/home/ubuntu/comfyUI2/ComfyUI/execution.py", line 81, in get_output_data
uis = []
File "/home/ubuntu/comfyUI2/ComfyUI/execution.py", line 74, in map_node_over_list
nodes.before_node_execution()
File "/home/ubuntu/comfyUI2/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 326, in apply_ipadapter
clip_embed = clip_embed.penultimate_hidden_states
getting KeyError: 'cond_or_uncond' on KSampler
Any idea what's causing this? I've attached my workflow.
2023-09-29T01:53:25.764492747Z File "/workspace/ComfyUI/comfy/k_diffusion/external.py", line 98, in predict_eps_discrete_timestep
2023-09-29T01:53:25.764493767Z return (input - self(input, sigma, **kwargs)) / utils.append_dims(sigma, input.ndim)
2023-09-29T01:53:25.764494887Z File "/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in call_impl
2023-09-29T01:53:25.764496097Z return forward_call(*args, **kwargs)
2023-09-29T01:53:25.764497087Z File "/workspace/ComfyUI/comfy/k_diffusion/external.py", line 125, in forward
2023-09-29T01:53:25.764498057Z eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
2023-09-29T01:53:25.764499137Z File "/workspace/ComfyUI/comfy/k_diffusion/external.py", line 151, in get_eps
2023-09-29T01:53:25.764500137Z return self.inner_model.apply_model(*args, **kwargs)
2023-09-29T01:53:25.764501107Z File "/workspace/ComfyUI/comfy/samplers.py", line 311, in apply_model
2023-09-29T01:53:25.764502207Z out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
2023-09-29T01:53:25.764503927Z File "/workspace/ComfyUI/comfy/samplers.py", line 289, in sampling_function
2023-09-29T01:53:25.764505047Z cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
2023-09-29T01:53:25.764506097Z File "/workspace/ComfyUI/comfy/samplers.py", line 263, in calc_cond_uncond_batch
2023-09-29T01:53:25.764507207Z output = model_function(input_x, timestep, **c).chunk(batch_chunks)
2023-09-29T01:53:25.764508227Z File "/workspace/ComfyUI/comfy/model_base.py", line 63, in apply_model
2023-09-29T01:53:25.764509277Z return self.diffusion_model(xc, t, context=context, y=c_adm, control=control, transformer_options=transformer_options).float()
2023-09-29T01:53:25.764510547Z File "/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-09-29T01:53:25.764511547Z return forward_call(*args, **kwargs)
2023-09-29T01:53:25.764512547Z File "/workspace/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 626, in forward
2023-09-29T01:53:25.764513557Z h = forward_timestep_embed(module, h, emb, context, transformer_options)
2023-09-29T01:53:25.764514557Z File "/workspace/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 56, in forward_timestep_embed
2023-09-29T01:53:25.764515807Z x = layer(x, context, transformer_options)
2023-09-29T01:53:25.764516897Z File "/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-09-29T01:53:25.764517967Z return forward_call(*args, **kwargs)
2023-09-29T01:53:25.764518977Z File "/workspace/ComfyUI/comfy/ldm/modules/attention.py", line 694, in forward
2023-09-29T01:53:25.764529747Z x = block(x, context=context[i], transformer_options=transformer_options)
2023-09-29T01:53:25.764530877Z File "/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-09-29T01:53:25.764531867Z return forward_call(*args, **kwargs)
2023-09-29T01:53:25.764532917Z File "/workspace/ComfyUI/comfy/ldm/modules/attention.py", line 526, in forward
2023-09-29T01:53:25.764533957Z return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint)
2023-09-29T01:53:25.764535097Z File "/workspace/ComfyUI/comfy/ldm/modules/diffusionmodules/util.py", line 123, in checkpoint
2023-09-29T01:53:25.764536267Z return func(*inputs)
2023-09-29T01:53:25.764537367Z File "/workspace/ComfyUI/comfy/ldm/modules/attention.py", line 623, in _forward
2023-09-29T01:53:25.764538367Z n = attn2_replace_patch[block_attn2](n, context_attn2, value_attn2, extra_options)
2023-09-29T01:53:25.764539387Z File "/workspace/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 238, in call
2023-09-29T01:53:25.764540477Z cond_or_uncond = outer_frame.f_locals["transformer_options"]["cond_or_uncond"]
2023-09-29T01:53:25.764541577Z KeyError: 'cond_or_uncond'
2023-09-29T01:53:25.764542737Z
2023-09-29T01:53:25.764925806Z Prompt executed in 1.38 seconds
RuntimeError: The size of tensor a (6144) must match the size of tensor b (13824) at non-singleton dimension 1
I realize this is still alpha, but wanted to report this issue if you weren't already aware. Feel free to close if so.
When I try to set a mask for IPAdapterApply node, I get this error when running:
File "/ComfyUI/comfy/samplers.py", line 289, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
File "/ComfyUI/comfy/samplers.py", line 263, in calc_cond_uncond_batch
output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
File "/ComfyUI/comfy/model_base.py", line 63, in apply_model
return self.diffusion_model(xc, t, context=context, y=c_adm, control=control, transformer_options=transformer_options).float()
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 626, in forward
h = forward_timestep_embed(module, h, emb, context, transformer_options)
File "/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 56, in forward_timestep_embed
x = layer(x, context, transformer_options)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/ComfyUI/comfy/ldm/modules/attention.py", line 693, in forward
x = block(x, context=context[i], transformer_options=transformer_options)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/ComfyUI/comfy/ldm/modules/attention.py", line 525, in forward
return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint)
File "/ComfyUI/comfy/ldm/modules/diffusionmodules/util.py", line 123, in checkpoint
return func(*inputs)
File "/ComfyUI/comfy/ldm/modules/attention.py", line 622, in _forward
n = attn2_replace_patch[block_attn2](n, context_attn2, value_attn2, extra_options)
File "/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 169, in __call__
ip_out = ip_out * mask_downsample
RuntimeError: The size of tensor a (6144) must match the size of tensor b (13824) at non-singleton dimension 1
'ClipVisionModel' object has no attribute 'processor'
I got this error this morning after updating Comfy
Error occurred when executing IPAdapterApply:
'ClipVisionModel' object has no attribute 'processor'
File "C:\Users\guill\Documents\ai\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\Users\guill\Documents\ai\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\Users\guill\Documents\ai\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\Users\guill\Documents\ai\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 326, in apply_ipadapter
clip_embed_zeroed = zeroed_hidden_states(clip_vision, image.shape[0])
File "C:\Users\guill\Documents\ai\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 99, in zeroed_hidden_states
inputs = clip_vision.processor(images=img, return_tensors="pt")
I think a commit in ComfyUI (ComfyUI@723847f6b3d5da21e5d712bc0139fb7197ba60a4) has caused this error when using ipadapterplus because self.processor was deleted from ClipVisionModel class
No matter what I do, I can't get to use the SDXL ones that require the 1.5 clipvision
Error occurred when executing IPAdapter:
Error(s) in loading state_dict for Resampler:
size mismatch for latents: copying a param with shape torch.Size([1, 16, 1280]) from checkpoint, the shape in current model is torch.Size([1, 16, 2048]).
size mismatch for proj_in.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 1280]).
size mismatch for proj_in.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for proj_out.weight: copying a param with shape torch.Size([2048, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 2048]).
size mismatch for layers.0.0.norm1.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.norm1.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.norm2.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.norm2.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.to_q.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([768, 2048]).
size mismatch for layers.0.0.to_kv.weight: copying a param with shape torch.Size([2560, 1280]) from checkpoint, the shape in current model is torch.Size([1536, 2048]).
size mismatch for layers.0.0.to_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 768]).
size mismatch for layers.0.1.0.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.1.0.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.1.1.weight: copying a param with shape torch.Size([5120, 1280]) from checkpoint, the shape in current model is torch.Size([8192, 2048]).
size mismatch for layers.0.1.3.weight: copying a param with shape torch.Size([1280, 5120]) from checkpoint, the shape in current model is torch.Size([2048, 8192]).
size mismatch for layers.1.0.norm1.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.0.norm1.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.0.norm2.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.0.norm2.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.0.to_q.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([768, 2048]).
size mismatch for layers.1.0.to_kv.weight: copying a param with shape torch.Size([2560, 1280]) from checkpoint, the shape in current model is torch.Size([1536, 2048]).
size mismatch for layers.1.0.to_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 768]).
size mismatch for layers.1.1.0.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.1.0.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.1.1.1.weight: copying a param with shape torch.Size([5120, 1280]) from checkpoint, the shape in current model is torch.Size([8192, 2048]).
size mismatch for layers.1.1.3.weight: copying a param with shape torch.Size([1280, 5120]) from checkpoint, the shape in current model is torch.Size([2048, 8192]).
size mismatch for layers.2.0.norm1.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.0.norm1.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.0.norm2.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.0.norm2.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.0.to_q.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([768, 2048]).
size mismatch for layers.2.0.to_kv.weight: copying a param with shape torch.Size([2560, 1280]) from checkpoint, the shape in current model is torch.Size([1536, 2048]).
size mismatch for layers.2.0.to_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 768]).
size mismatch for layers.2.1.0.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.1.0.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.2.1.1.weight: copying a param with shape torch.Size([5120, 1280]) from checkpoint, the shape in current model is torch.Size([8192, 2048]).
size mismatch for layers.2.1.3.weight: copying a param with shape torch.Size([1280, 5120]) from checkpoint, the shape in current model is torch.Size([2048, 8192]).
size mismatch for layers.3.0.norm1.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.0.norm1.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.0.norm2.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.0.norm2.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.0.to_q.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([768, 2048]).
size mismatch for layers.3.0.to_kv.weight: copying a param with shape torch.Size([2560, 1280]) from checkpoint, the shape in current model is torch.Size([1536, 2048]).
size mismatch for layers.3.0.to_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 768]).
size mismatch for layers.3.1.0.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.1.0.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.3.1.1.weight: copying a param with shape torch.Size([5120, 1280]) from checkpoint, the shape in current model is torch.Size([8192, 2048]).
size mismatch for layers.3.1.3.weight: copying a param with shape torch.Size([1280, 5120]) from checkpoint, the shape in current model is torch.Size([2048, 8192]).
File "F:\AI_repos\Stable_Comfy_UI\ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "F:\AI_repos\Stable_Comfy_UI\ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "F:\AI_repos\Stable_Comfy_UI\ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "F:\AI_repos\Stable_Comfy_UI\ComfyUI_windows_portable\ComfyUI\custom_nodes\IPAdapter-ComfyUI\ip_adapter.py", line 163, in adapter
self.ipadapter = IPAdapterModel(
File "F:\AI_repos\Stable_Comfy_UI\ComfyUI_windows_portable\ComfyUI\custom_nodes\IPAdapter-ComfyUI\ip_adapter.py", line 103, in __init__
self.image_proj_model.load_state_dict(state_dict["image_proj"])
File "F:\AI_repos\Stable_Comfy_UI\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(`
I always get this error, I even tried the workflow that you have posted for the SDXL model, but nothing makes this work it's driving me nuts. All the other ipadapters work fine. But the ones with vit-h that require the 1.5 image encoder, always print me this error.
Please help, I'd appreciate it a lot.
Thanks in advance. Also, you're doing fantastic work with Ipadapter. Thank you for your contribution
a size mismatching error
Does anybody know how to fix it?
`Error(s) in loading state_dict for ImageProjModel:
size mismatch for proj.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
File "D:\AI\cmfui\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\AI\cmfui\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\AI\cmfui\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\AI\cmfui\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 277, in apply_ipadapter
self.ipadapter = IPA(
File "D:\AI\cmfui\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 128, in init
self.image_proj_model.load_state_dict(ipadapter_model["image_proj"])
File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict`
ip-adapter-plus_sd15.bin, ip-adapter-plus-face.bin throw
Hello,
was trying this custom node, selecting ip-adapter_sd15 and ip-adapter_sd15_light bins works great, though the other two throw the following to console:
got prompt
INFO: the IPAdapter reference image is not a square, CLIPImageProcessor will resize and crop it at the center. If the main focus of the picture is not in the middle the result might not be what you are expecting.
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\PIL\Image.py", line 3089, in fromarray
mode, rawmode = _fromarray_typemap[typekey]
KeyError: ((1, 1, 224, 3), '|u1')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 325, in apply_ipadapter
clip_embed_zeroed = zeroed_hidden_states(clip_vision, image.shape[0])
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 98, in zeroed_hidden_states
inputs = clip_vision.processor(images=img, return_tensors="pt")
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\image_processing_utils.py", line 446, in call
return self.preprocess(images, **kwargs)
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\models\clip\image_processing_clip.py", line 318, in preprocess
images = [self.resize(image=image, size=size, resample=resample) for image in images]
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\models\clip\image_processing_clip.py", line 318, in
images = [self.resize(image=image, size=size, resample=resample) for image in images]
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\models\clip\image_processing_clip.py", line 144, in resize
return resize(image, size=output_size, resample=resample, data_format=data_format, **kwargs)
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\image_transforms.py", line 275, in resize
image = to_pil_image(image)
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\transformers\image_transforms.py", line 163, in to_pil_image
return PIL.Image.fromarray(image)
File "D:\WORK\conda_envs\ConfyUI\ComfyUI_windows_portable\python_embeded\lib\site-packages\PIL\Image.py", line 3092, in fromarray
raise TypeError(msg) from e
TypeError: Cannot handle this data type: (1, 1, 224, 3), |u1
Maybe doing something ill advised on my side, maybe some edge case, either way if someone has a clue why those two don't work, would appreciate directions.
Running with DirectML gives "RuntimeError: The parameter is incorrect."
I'm running ComfyUI with --directml and using the example workflow from the readme.
There is some issue with batch size, I get "RuntimeError: The parameter is incorrect." here. Stack trace below.
It looks like b = n.shape[0] = 1, so 0 is passed to repeat(b//2, ...). I guess the expected value would be b = 2 but not really sure why.
Setting batch size to 2 will no longer trigger the error, produces 2 images, but only the second one is affected by IP adapter. Also when passing b to repeat instead of b//2. Any ideas?
Additional notes:
torch.autocast("cuda")is maybe also a problem, but in my case everything is float32- everything works fine with CUDA or CPU
Stack:
Traceback (most recent call last):
File "C:\Dev\DML_Comfy\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\Dev\DML_Comfy\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\Dev\DML_Comfy\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\Dev\DML_Comfy\ComfyUI\nodes.py", line 1236, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "C:\Dev\DML_Comfy\ComfyUI\nodes.py", line 1206, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "C:\Dev\DML_Comfy\ComfyUI\comfy\sample.py", line 93, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\samplers.py", line 742, in sample
samples = getattr(k_diffusion_sampling, "sample_{}".format(self.sampler))(self.model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar)
File "C:\Dev\DML_Comfy\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\k_diffusion\sampling.py", line 580, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "C:\Dev\DML_Comfy\python\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\samplers.py", line 323, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed)
File "C:\Dev\DML_Comfy\python\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\k_diffusion\external.py", line 125, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\k_diffusion\external.py", line 151, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\samplers.py", line 311, in apply_model
out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\samplers.py", line 289, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\samplers.py", line 265, in calc_cond_uncond_batch
output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\model_base.py", line 63, in apply_model
return self.diffusion_model(xc, t, context=context, y=c_adm, control=control, transformer_options=transformer_options).float()
File "C:\Dev\DML_Comfy\python\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\ldm\modules\diffusionmodules\openaimodel.py", line 626, in forward
h = forward_timestep_embed(module, h, emb, context, transformer_options)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\ldm\modules\diffusionmodules\openaimodel.py", line 56, in forward_timestep_embed
x = layer(x, context, transformer_options)
File "C:\Dev\DML_Comfy\python\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\ldm\modules\attention.py", line 693, in forward
x = block(x, context=context[i], transformer_options=transformer_options)
File "C:\Dev\DML_Comfy\python\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\ldm\modules\attention.py", line 525, in forward
return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\ldm\modules\diffusionmodules\util.py", line 123, in checkpoint
return func(*inputs)
File "C:\Dev\DML_Comfy\ComfyUI\comfy\ldm\modules\attention.py", line 622, in _forward
n = attn2_replace_patch[block_attn2](n, context_attn2, value_attn2, extra_options)
File "C:\Dev\DML_Comfy\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 188, in __call__
uncond_cond = torch.cat([uncond.repeat(b//2, 1, 1), cond.repeat(b//2, 1, 1)], dim=0)
RuntimeError: The parameter is incorrect.
'ClipVisionModel' object has no attribute 'processor'
After the latest update of ComfyUI I get this error message while running the "Load IPAdapter" node:
Requested to load CLIPVisionModelWithProjection
Loading 1 new model
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "e:\Stablediffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "e:\Stablediffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "e:\Stablediffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 331, in apply_ipadapter
clip_embed_zeroed = zeroed_hidden_states(clip_vision, image.shape[0])
File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 103, in zeroed_hidden_states
inputs = clip_vision.processor(images=img, return_tensors="pt")
AttributeError: 'ClipVisionModel' object has no attribute 'processor'
Prompt executed in 1.42 seconds
Could you please check?
Can't get a workflow with SDXL working. Have an example to share?
Me and another person have tried modifying the 1.5 example workflows shared in this repo to work with SDXL. We're getting the black square output, not matter what checkpoint, vae, or fp16 on or off we use. Do you have an example of a working one that we can try to use for troubleshooting?
SDXL usage?
Are you able to provide a workflow specifically for SDXL please?
Finding the instructions a bit vague, I've downloaded the models as instructed but it's a bit unclear as to specfically how we're plugging this together. Currently I have this setup -
I've had no other option other than to rename the linked clip-vision model (H) to ckpt because the bin file refuses to show up in the UI.
Running like this results in -
Very much appreciated if you'd be able to assist.
Rounding issue?
Hi,
I did not look into it deeply and don't know IPAdapter much so maybe there is another reason but it seems like there is a rounding issue in the weights blending
will_musk_webm.mp4
[Not an issue] My xp with the noise option
As you are asking for feedback in your readme, I did several test with the noise option. In my case, it makes the image closer to what I ask in the prompt without having to reduce the weight of the ipadapter.
For example:
grim reaper origami, minimalist background, 8k photo wallpaper
- CFG 5.5
- CFG 9.0
Origami and minimalist background are clearly more prominent in the "1" version. The Ipadapter weight is at 0.35.
I wonder if you should not change the name though. Maybe inspired by ControlNet, something like "my prompt is more important"? And just keep a switch with 0 and 1 values applied?
README.md typo with models path
Greetings! Linux user here. This one tripped me up for a second...
Small typo in README.md:
ComfyUI/custom_nodes/ComfyUI_IPAdapter_Plus/models should be ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/models
Unable to install the latest version
Hello,
I have the version from October 12, 2023, which works without any problems.
However, I'm unable to install the update (ConfyUI indicates that it wasn't loaded).
I've completely uninstalled IPA+ with the Manager and then tried to reinstall it, but without success.
What information do you need to resolve this issue?
No Longer Working with AnimateDiffLoader
Previously worked fine but now all of my image outputs a covered in noise. I am unable to upload a workflow.json so I renamed to .txt file. You can change to .json to see
Also included screenshots of the node network + screenshot of the output.
size mismatch for proj_in.weight:
Hello - the repo works great when using the base ipadapter_sd15.bin model - however when I move up to the plus model without changing anything else in my workflow, I get the following error:
`Error occurred when executing IPAdapterApply:
Error(s) in loading state_dict for Resampler:
size mismatch for proj_in.weight: copying a param with shape torch.Size([768, 1280]) from checkpoint, the shape in current model is torch.Size([768, 1024]).
File "/home/-user/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/home/-user/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/home/-user/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/home/-user/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 237, in apply_ipadapter
self.ipadapter = IPAdapterPlus(
File "/home/-user/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 90, in init
self.image_proj_model.load_state_dict(self.ipadapter_model["image_proj"])
File "/home/-user/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
`
Just wondering if you have any idea or if this is a known issue
[Request] About the model placement location
Could you include a mechanism, like Comfy's yaml, that allows us to change to a specified location?
Ideally, it would be easier to use it here as a default.
ComfyUI/models/controlnet
SDXL "step must be nonzero" error with batch 1 only
I got a strange error when using SDXL (with ip-adapter-_sdxl-vit-h.bin or ip-adapter-_sdxl.bin), if I use a size larger than 768x768 (in my example 960x1152) I got a "step must be nonzero" in KSampler or KSamplerAdvanced node, but only if the batch size is odd. If I set the batch size to 2 or 4, it get rid of the error, but the odd image is affected by the noise and not the IPadapter, and the even images are affected by the IPadapter, but not the noise setting.
An output containing the workflow :
The stack trace :
Error occurred when executing KSamplerAdvanced:
step must be nonzero
File "F:\ComfyUI\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "F:\ComfyUI\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "F:\ComfyUI\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "F:\ComfyUI\ComfyUI\nodes.py", line 1270, in sample
return common_ksampler(model, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise, disable_noise=disable_noise, start_step=start_at_step, last_step=end_at_step, force_full_denoise=force_full_denoise)
File "F:\ComfyUI\ComfyUI\nodes.py", line 1206, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 22, in informative_sample
raise e
File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample
return original_sample(*args, **kwargs)
File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 107, in animatediff_sample
return orig_comfy_sample(model, *args, **kwargs)
File "F:\ComfyUI\ComfyUI\comfy\sample.py", line 93, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "F:\ComfyUI\ComfyUI\comfy\samplers.py", line 742, in sample
samples = getattr(k_diffusion_sampling, "sample_{}".format(self.sampler))(self.model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar)
File "F:\ComfyUI\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "F:\ComfyUI\ComfyUI\comfy\k_diffusion\sampling.py", line 613, in sample_dpmpp_2m_sde
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "F:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "F:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "F:\ComfyUI\ComfyUI\comfy\samplers.py", line 323, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed)
File "F:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "F:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "F:\ComfyUI\ComfyUI\comfy\k_diffusion\external.py", line 125, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "F:\ComfyUI\ComfyUI\comfy\k_diffusion\external.py", line 151, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "F:\ComfyUI\ComfyUI\comfy\samplers.py", line 311, in apply_model
out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
File "F:\ComfyUI\ComfyUI\comfy\samplers.py", line 289, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
File "F:\ComfyUI\ComfyUI\comfy\samplers.py", line 265, in calc_cond_uncond_batch
output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
File "F:\ComfyUI\ComfyUI\comfy\model_base.py", line 63, in apply_model
return self.diffusion_model(xc, t, context=context, y=c_adm, control=control, transformer_options=transformer_options).float()
File "F:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "F:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "F:\ComfyUI\ComfyUI\custom_nodes\SeargeSDXL\modules\custom_sdxl_ksampler.py", line 70, in new_unet_forward
x0 = old_unet_forward(self, x, timesteps, context, y, control, transformer_options, **kwargs)
File "F:\ComfyUI\ComfyUI\comfy\ldm\modules\diffusionmodules\openaimodel.py", line 627, in forward
h = forward_timestep_embed(module, h, emb, context, transformer_options)
File "F:\ComfyUI\ComfyUI\comfy\ldm\modules\diffusionmodules\openaimodel.py", line 56, in forward_timestep_embed
x = layer(x, context, transformer_options)
File "F:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "F:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "F:\ComfyUI\ComfyUI\comfy\ldm\modules\attention.py", line 693, in forward
x = block(x, context=context[i], transformer_options=transformer_options)
File "F:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "F:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "F:\ComfyUI\ComfyUI\comfy\ldm\modules\attention.py", line 525, in forward
return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint)
File "F:\ComfyUI\ComfyUI\comfy\ldm\modules\diffusionmodules\util.py", line 123, in checkpoint
return func(*inputs)
File "F:\ComfyUI\ComfyUI\comfy\ldm\modules\attention.py", line 622, in _forward
n = attn2_replace_patch[block_attn2](n, context_attn2, value_attn2, extra_options)
File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 196, in __call__
ip_out = attention(q, ip_k, ip_v, extra_options)
File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 75, in attention
out = F.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=False)
An example of 4 batch outputs with noise 0.3 :
And with noise 0.0 :
As you can see only the first 2 images are affected by the noise, but not the adaptation. And the last 2 images are correctly adapted, but the noise doesn't do anything.
I'm using a 3070Ti with 8GB VRAM, and ComfyUI with --bf16-vae --disable-xformers --use-pytorch-cross-attention command line options, and torch 2.1.0.dev+cu118 (for the bf16 VAE) manually installed.
The noise effect on batch 2/4 is also affecting SD 1.5, so it may be a separate bug :
But with SD 1.5 I can render an image > 1024x1024 without the "step must be nonzero" bug, even if it is not a very useful resolution for this generation of models.
Keep getting error from appy ipAdapter
'CLIPVisionModelOutput' object has no attribute 'penultimate_hidden_states'
File "C:\Users\SD\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\Users\SD\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\Users\SD\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\Users\SD\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 326, in apply_ipadapter
clip_embed = clip_embed.penultimate_hidden_states
Hello sdxl. A black picture appears in the model, how can I solve it?
Hello sdxl. A black picture appears in the model, how can I solve it?
Size mismatch for proj.weight
Hello. Was trying the workflow included in this repo but keep getting this error:
!!! Exception during processing !!!
Traceback (most recent call last):
File "F:\ComfyUI-Update-9-1-2023\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "F:\ComfyUI-Update-9-1-2023\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "F:\ComfyUI-Update-9-1-2023\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "F:\ComfyUI-Update-9-1-2023\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 245, in apply_ipadapter
self.ipadapter = IPAdapter(
File "F:\ComfyUI-Update-9-1-2023\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 90, in __init__
self.image_proj_model.load_state_dict(self.ipadapter_model["image_proj"])
File "F:\ComfyUI-Update-9-1-2023\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ImageProjModel:
size mismatch for proj.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
Any ideas? Thanks!
IPadapter Plus using signficantly more VRAM since latest ComfyUI Update
Hi I wanted to submit this. Finding significantly more VRAM usage since latest ComfyUI update (~4gb more vram on a image batch of 16).
Error occurred when executing IPAdapterApply
Error occurred when executing IPAdapterApply:
Error(s) in loading state_dict for ImageProjModel:
size mismatch for proj.weight: copying a param with shape torch.Size([8192, 1024]) from checkpoint, the shape in current model is torch.Size([8192, 1280]).
File "D:\AXAI\ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\AXAI\ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\AXAI\ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\AXAI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 342, in apply_ipadapter
self.ipadapter = IPA(
^^^^
File "D:\AXAI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 182, in init
self.image_proj_model.load_state_dict(ipadapter_model["image_proj"])
File "D:\AXAI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 2152, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
what models should put at CLIP Vesion Load?
from this example, it should put the model "IPAdapter_image_encoder_sd15.safetensors", where I find it? it's not CLIP base/large/big model here?
In ComfyUI, there's an error when using ip-adapter-plus_sdxl_vit-h.bin as IPAdapter model and sd1.5 image encoders
The workflow file is here:
Uploading ipadapter_error.zip…
and this is the model I use:
checkpoint: sd_xl_base1.0
IpAdapter model and image encoder mode:
Workflow that includes Refiner.
Could you explain or make a workflow that includes Refiner? When I try to include refiner, it hugely changes the style. Thanks!
Missing nodes
Missing nodes.. did everything per the installation directions - any fix?
Embeds save to image output directory
It looks like the code tries to load embeds from the input directory but saves them to the default image output directory, with a default "embeds/" dialogue box opening up in comfyui itself.
I symlinked the {default output}/embeds folder to input/embeds which fixed it, and changed
files = [f for f in os.listdir(f'{input_dir}/embeds/') if os.path.isfile(os.path.join(input_dir, f)) and f.endswith(".ipadpt")]
to
embedsdir = os.path.join(input_dir, 'embeds')
files = [f for f in os.listdir(embedsdir) if os.path.isfile(os.path.join(embedsdir, f)) and f.endswith(".ipadpt")]
Output failed inputs missing
I updated IPAdapter Plus, and redownloaded the adapter model, but I keep getting this error:
[REQUEST] - Weights for image inputs
Hey hey!!
Great work on this, very impressive for sure.
I'm not sure IP_Adapter works like this or not, but will ask anyways.
As a feature request, I'm wondering if it's possible to add controls or a node that weighs how much of an image contributes to a set of batch images. Example below:
Can't save and load IPadapter embeds
I don't know for sure if the problem is in the loading or the saving. When I set up a chain to save an embed from an image it executes okay. But the loader doesn't allow you to choose an embed that you (maybe) saved. It just has the embeds widget that says undefined, and you can't change it. Either way, the whole process doesn't work. I was expecting being able to save embeds for later, saving time by applying a ready embed.
Error 'contextlib' is not defined
Hello, I download all the necessary prerequisites and used one of the workflows included but everytime i run i get this error:
Error occurred when executing IPAdapterCLIPVisionEncode:
name 'contextlib' is not defined
File "F:\Comfy PC\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "F:\Comfy PC\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "F:\Comfy PC\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "F:\Comfy PC\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 181, in encode
output = clip_vision.encode_image_IPAdapter(image)
File "F:\Comfy PC\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 326, in encode_image_IPAdapter
with precision_scope(comfy.model_management.get_autocast_device(self.load_device), torch.float32):
File "F:\Comfy PC\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 324, in
precision_scope = lambda a, b: contextlib.nullcontext(a)
'ModuleList' object has no attribute '1' when trying to use canny example
!!! Exception during processing !!!
Traceback (most recent call last):
File "D:\Projects\ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\Projects\ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\Projects\ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\Projects\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1236, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "D:\Projects\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1206, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "D:\Projects\ComfyUI_windows_portable\ComfyUI\comfy\sample.py", line 93, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "D:\Projects\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 644, in sample
pre_run_control(self.model_wrap, negative + positive)
File "D:\Projects\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 488, in pre_run_control
x[1]['control'].pre_run(model.inner_model.inner_model, percent_to_timestep_function)
File "D:\Projects\ComfyUI_windows_portable\ComfyUI\comfy\controlnet.py", line 269, in pre_run
comfy.utils.set_attr(self.control_model, k, self.control_weights[k].to(dtype).to(comfy.model_management.get_torch_device()))
File "D:\Projects\ComfyUI_windows_portable\ComfyUI\comfy\utils.py", line 264, in set_attr
obj = getattr(obj, name)
File "D:\Projects\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1614, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'ModuleList' object has no attribute '1'
Tensor mis
When testing on comfyui, I get the following error:
"RuntimeError: Error(s) in loading state_dict for Resampler: "
"size mismatch for latents: copying a param with shape torch.Size([1, 16, 1280]) from checkpoint, the shape in current model is torch.Size([1, 16, 2048]).
size mismatch for proj_in.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 1664]).
size mismatch for proj_in.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for proj_out.weight: copying a param with shape torch.Size([2048, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 2048]).
size mismatch for layers.0.0.norm1.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.norm1.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.norm2.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.norm2.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2048]).
size mismatch for layers.0.0.to_q.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([768, 2048]).
size mismatch for layers.0.0.to_kv.weight: copying a param with shape torch.Size([2560, 1280]) from checkpoint, the shape in current model is torch.Size([1536, 2048]).
size mismatch for layers.0.0.to_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([2048, 768])."
Do you have any suggestions on how to fix this?
high VRAM usage with large batches or resolutions
Hi, thanks for this great repo,
I have been running into vram issue on my 4090 when trying to use IPadapter with anything greater than batch size 16 at 512x768.
This is mostly an issue in Animate diff, where it would be nice to generate larger batches or even at larger sizes.
Wondering if this is expected memory usage for IPadapter? If so, please feel free to close this issue.
Here's my simple workflow for reproducing the issue and the error message for out of memory below:
Error occurred when executing KSampler:
Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated : 21.72 GiB
Requested : 720.00 MiB
Device limit : 23.65 GiB
Free (according to CUDA): 29.69 MiB
PyTorch limit (set by user-supplied memory fraction)
: 17179869184.00 GiB
File "/workspace/ComfyUI/execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/workspace/ComfyUI/execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/workspace/ComfyUI/execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/workspace/ComfyUI/nodes.py", line 1236, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/workspace/ComfyUI/nodes.py", line 1206, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/workspace/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 107, in animatediff_sample
return orig_comfy_sample(model, *args, **kwargs)
File "/workspace/ComfyUI/comfy/sample.py", line 97, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/workspace/ComfyUI/comfy/samplers.py", line 785, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler(), sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/workspace/ComfyUI/comfy/samplers.py", line 690, in sample
samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "/workspace/ComfyUI/comfy/samplers.py", line 630, in sample
samples = getattr(k_diffusion_sampling, "sample_{}".format(sampler_name))(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **extra_options)
File "/workspace/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/workspace/ComfyUI/comfy/k_diffusion/sampling.py", line 137, in sample_euler
denoised = model(x, sigma_hat * s_in, **extra_args)
File "/workspace/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/ComfyUI/comfy/samplers.py", line 323, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed)
File "/workspace/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/ComfyUI/comfy/k_diffusion/external.py", line 125, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "/workspace/ComfyUI/comfy/k_diffusion/external.py", line 151, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "/workspace/ComfyUI/comfy/samplers.py", line 311, in apply_model
out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
File "/workspace/ComfyUI/comfy/samplers.py", line 289, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
File "/workspace/ComfyUI/comfy/samplers.py", line 265, in calc_cond_uncond_batch
output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
File "/workspace/ComfyUI/comfy/model_base.py", line 63, in apply_model
return self.diffusion_model(xc, t, context=context, y=c_adm, control=control, transformer_options=transformer_options).float()
File "/workspace/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 659, in forward
h = forward_timestep_embed(module, h, emb, context, transformer_options, output_shape)
File "/workspace/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 56, in forward_timestep_embed
x = layer(x, context, transformer_options)
File "/workspace/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/ComfyUI/comfy/ldm/modules/attention.py", line 695, in forward
x = block(x, context=context[i], transformer_options=transformer_options)
File "/workspace/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/ComfyUI/comfy/ldm/modules/attention.py", line 525, in forward
return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint)
File "/workspace/ComfyUI/comfy/ldm/modules/diffusionmodules/util.py", line 123, in checkpoint
return func(*inputs)
File "/workspace/ComfyUI/comfy/ldm/modules/attention.py", line 635, in _forward
x = self.ff(self.norm3(x)) + x
File "/workspace/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/ComfyUI/comfy/ldm/modules/attention.py", line 82, in forward
return self.net(x)
File "/workspace/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/workspace/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/ComfyUI/comfy/ldm/modules/attention.py", line 62, in forward
return x * F.gelu(gate)
cannot import name 'clip_preprocess' from 'comfy.clip_vision
Hello import failed since today update on mac osx
When should we be using ip_adapter vs ip_adapter plus?
Hi! Thanks for making this extension--much appreciated. 2 questions:
- What is the difference between ip_adapter and ip_adapter plus?
- Also, for the image encoders--what are their actual values? I.e, I'm currently using open_clip_pytorch (which I'm guessing is a ViT-H) for the SD1.5 image encoder. But what is the image encoder for SDXL?
RuntimeError: Error(s) in loading state_dict for ImageProjModel:
I have tried with the example workflow which also produce this error.
All nodes and comfy are updated.
RuntimeError: Error(s) in loading state_dict for ImageProjModel:
size mismatch for proj.weight: copying a param with shape torch.Size([8192, 1024]) from checkpoint, the shape in current model is torch.Size([8192, 1280]).
Error occurred when executing IPAdapterApply:
when i run the example of IPAdapter_inpaint.json , it comes error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.