Bug when using with controlnet about comfyui-tileddiffusion HOT 15 OPEN

I managed to find a quick fix to the multiple controlnet problem in the links I had shared above (https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/tiled_diffusion.py#L308-L314). Donot know of a solution for instant ID yet.

If you just want to use tile diffusion with multiple controlnets for now, you can use the fork and branch from the link.

But i get errors on 3 Control Nets now xD

2 works. 3 not :D

from comfyui-tileddiffusion.

Hi, sorry for the delayed reply. I made a few more changes in the same fork. It should now atleast work with 3 controlnets for SDXL (I tried with full canny, depth and TTPlanet tile). It was a minor bug in my code.

from comfyui-tileddiffusion.

I was able to use it with 3 controlnets, can you check you are using the correct branch in my fork? You have can use "git switch fix_controlnet".

from comfyui-tileddiffusion.

I managed to run this workflow:

https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/tile_diffusion_sr.json

on G5.2x large aws machine (24 gb gpu memory, ubuntu 22.04 native os python with venv) with the packages:
https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/versions

The xformers seem to help.

from comfyui-tileddiffusion.

Thanks for reporting this! I ran into the same problem a week ago and hadn't had a chance to file an issue just yet. But I can also confirm that the node breaks when using more than one ControlNet in series.

from comfyui-tileddiffusion.

I ran into these bugs when working with instant ID on huge images. Even after fixing this using the method I have mentioned, the issues remained on instant ID side. So once we have a proper solution here, I will report the issue in instant ID page!

from comfyui-tileddiffusion.

Any idea to fix it ?

from comfyui-tileddiffusion.

I managed to find a quick fix to the multiple controlnet problem in the links I had shared above (https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/tiled_diffusion.py#L308-L314). Donot know of a solution for instant ID yet.

If you just want to use tile diffusion with multiple controlnets for now, you can use the fork and branch from the link.

from comfyui-tileddiffusion.

I managed to find a quick fix to the multiple controlnet problem in the links I had shared above (https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/tiled_diffusion.py#L308-L314). Donot know of a solution for instant ID yet.

If you just want to use tile diffusion with multiple controlnets for now, you can use the fork and branch from the link.

Duuuude Youre crazy! Thank you very much !

Can you text me on discord ?
Would like to share my upscale workflow with you as thank you!

Dervlex is my name on Discord

from comfyui-tileddiffusion.

!!! Exception during processing!!! object of type 'NoneType' has no len()
Traceback (most recent call last):
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\nodes.py", line 1344, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\nodes.py", line 1314, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample
return original_sample(*args, **kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_reference.py", line 47, in refcn_sample
return orig_comfy_sample(model, *args, **kwargs)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\sample.py", line 37, in sample
samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 755, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 657, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 644, in sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 623, in inner_sample
samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 534, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
File "C:\Users\dervl\ComfyUI Neu\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\k_diffusion\sampling.py", line 745, in sample_lcm
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 272, in call
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 610, in call
return self.predict_noise(*args, **kwargs)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 613, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 258, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion.patches.py", line 89, in calc_cond_batch
output = model_options['model_function_wrapper'](model.apply_model, {"input": input_x, "timestep": timestep_, "c": c, "cond_or_uncond": cond_or_uncond}).chunk(batch_chunks)
File "C:\Users\dervl\ComfyUI Neu\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion\tiled_diffusion.py", line 554, in call
self.process_controlnet(x_tile.shape, x_tile.dtype, c_in, cond_or_uncond, bboxes, N, batch_id)
File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion\tiled_diffusion.py", line 312, in process_controlnet
while len(self.control_params[tuple_key][param_id]) <= batch_id:
TypeError: object of type 'NoneType' has no len()

from comfyui-tileddiffusion.

I managed to find a quick fix to the multiple controlnet problem in the links I had shared above (https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/tiled_diffusion.py#L308-L314). Donot know of a solution for instant ID yet.
If you just want to use tile diffusion with multiple controlnets for now, you can use the fork and branch from the link.

But i get errors on 3 Control Nets now xD

2 works. 3 not :D

Same. Just tested with 4. Doesn't work, but if I disable 2 of them then it will work. Nothing more then 2 will work.

from comfyui-tileddiffusion.

Hi, sorry for the delayed reply. I made a few more changes in the same fork. It should now atleast work with 3 controlnets for SDXL (I tried with full canny, depth and TTPlanet tile). It was a minor bug in my code.

I tried your new update with more then one controlnet and it wouldn't work. Disabling everything except one controlnet makes it work.

from comfyui-tileddiffusion.

!!! Exception during processing!!! object of type 'NoneType' has no len() Traceback (most recent call last): File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\nodes.py", line 1344, in sample return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\nodes.py", line 1314, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample return original_sample(*args, **kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations. File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_reference.py", line 47, in refcn_sample return orig_comfy_sample(model, *args, **kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\sample.py", line 37, in sample samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 755, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 657, in sample return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 644, in sample output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 623, in inner_sample samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 534, in sample samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options) File "C:\Users\dervl\ComfyUI Neu\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\k_diffusion\sampling.py", line 745, in sample_lcm denoised = model(x, sigmas[i] * s_in, **extra_args) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 272, in call out = self.inner_model(x, sigma, model_options=model_options, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 610, in call return self.predict_noise(*args, **kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 613, in predict_noise return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 258, in sampling_function out = calc_cond_batch(model, conds, x, timestep, model_options) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion.patches.py", line 89, in calc_cond_batch output = model_options['model_function_wrapper'](model.apply_model, {"input": input_x, "timestep": timestep_, "c": c, "cond_or_uncond": cond_or_uncond}).chunk(batch_chunks) File "C:\Users\dervl\ComfyUI Neu\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion\tiled_diffusion.py", line 554, in call self.process_controlnet(x_tile.shape, x_tile.dtype, c_in, cond_or_uncond, bboxes, N, batch_id) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion\tiled_diffusion.py", line 312, in process_controlnet while len(self.control_params[tuple_key][param_id]) <= batch_id: TypeError: object of type 'NoneType' has no len()

Hi, what was the video memory you have ? Can you try reducing the tile diffusion batch size?

from comfyui-tileddiffusion.

Hi, I also noticed that this error is very difficult to reproduce and depends on the resolution and other matters of the input image. I found that bypassing instant id resolves the error for me. Probably it comes from combination of insight face (face detection) and other complex interactions from instant id.

from comfyui-tileddiffusion.

@aravindhv10 I am testing the update now. So far its working very great! I noticed some incompatibility with midas depth node, and zoe depth node, but it works with the regular depth node and leres node.

from comfyui-tileddiffusion.