vinairesearch / blur-kernel-space-exploring Goto Github PK

In Readme, the author said “To do image deblurring, data augmentation, and blur generation, you first need to train the blur encoding network (The F function in the paper). This is the only network that you need to train.“ However, after I retrained the F function using out-of-focus blur- sharp image pairs and used it in the image deblurring script "generic_deblur.py" to deblur out-of-focus images, the results are not yet satisfying.

I trained with a batch_size of 13 and iteration of 600,000. The final loss is around 1e03, and the training datasets volume is around 3000 image pairs.

I wonder if I need to retrain the G function as well, if I am using the network to deblur out-of-focus images. If needed, is it possible to provide a script that trains both G and F functions?

Thank you!

I couldn't download pretrained models, the links are invalid. Could you update the link? thank you!

Thank you for your excellent work!
How much time did your work take to complete the training process on the v100gpu

Thank you for this repo is really awesome. But I have a small question:
how is the "warmup_k_path" file created, what is it used for in the deblur process.

I have updated the download links of the pretrained models. Please check again.
Sorry for the inconvenience.

Originally posted by @P0lyFish in #8 (comment)

first of all, impressive work!

1. I am confused about the form of kernel. Ki is the kernel of uniform blur or can also be kernel of non-uniform blur?
2. Why the regularization term on the L2 norm of the kernel k works?

I run in command line python domain_specific_deblur.py --input_dir imgs/blur_faces
--output_dir experiments/domain_specific_deblur/results
--yml_path options/domain_specific_deblur/stylegan.yml
and using 2 GPU, but I have error so How do I can fix this?

The size of the image which I used is 640*640. It can be diviable by 128. Thanks for your help!

Sample #0/10
Traceback (most recent call last):
File ".\generate_blur.py", line 53, in
main()
File ".\generate_blur.py", line 45, in main
LQ_tensor = model.adaptKernel(HQ_tensor, kernel)
File "D:\Project\Pycharm\blur-kernel-space-exploring-main\models\kernel_encoding\kernel_wizard.py", line 157, in adaptKernel
out = self.adapter(x_sharp, kernel)
File "D:\ProgramData\Anaconda3\envs\py38\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Project\Pycharm\blur-kernel-space-exploring-main\models\kernel_encoding\kernel_wizard.py", line 99, in forward
return self.model(x, k)
File "D:\ProgramData\Anaconda3\envs\py38\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Project\Pycharm\blur-kernel-space-exploring-main\models\backbones\unet_parts.py", line 103, in forward
return self.up(self.submodule(self.down(x), noise))
File "D:\ProgramData\Anaconda3\envs\py38\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Project\Pycharm\blur-kernel-space-exploring-main\models\backbones\unet_parts.py", line 109, in forward
return torch.cat((self.up(self.submodule(self.down(x), noise)), x), dim=1)
File "D:\ProgramData\Anaconda3\envs\py38\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Project\Pycharm\blur-kernel-space-exploring-main\models\backbones\unet_parts.py", line 109, in forward
return torch.cat((self.up(self.submodule(self.down(x), noise)), x), dim=1)
File "D:\ProgramData\Anaconda3\envs\py38\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Project\Pycharm\blur-kernel-space-exploring-main\models\backbones\unet_parts.py", line 109, in forward
return torch.cat((self.up(self.submodule(self.down(x), noise)), x), dim=1)
File "D:\ProgramData\Anaconda3\envs\py38\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Project\Pycharm\blur-kernel-space-exploring-main\models\backbones\unet_parts.py", line 107, in forward
return torch.cat((self.up(torch.cat((self.down(x), noise), dim=1)), x), dim=1)
RuntimeError: Sizes of tensors must match except in dimension 2. Got 2 and 5 (The offending index is 0)

Hello! Thank you so much for sharing the code and the model.

I would like to ask, what protocol is the pre-training model? Can I use it in a business project?

Your reply is very much needed.

Hi, I'm trying to run domain_specific_deblur.py with styleGAN2 on 512x512 dimension with pretrained models. However, after I fixed the latent and noise dimentions in dsd_stylegan2.py, the code still errors out on unet_parts.py. Is 512x512 supported?

The stacktrace is as below:

Traceback (most recent call last):
  File "domain_specific_deblur.py", line 84, in <module>
    for j, (HR, LR) in enumerate(model(ref_im)):
  File "/deblur/blur-kernel-space-exploring/models/dsd/dsd.py", line 188, in forward
    self.optimize_x_step(epoch)
  File "/deblur/blur-kernel-space-exploring/models/dsd/dsd.py", line 150, in optimize_x_step
    loss, loss_dict = self.loss_builder(latent_in, self.gen_im, self.gen_ker, epoch)
  File "//anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/deblur/blur-kernel-space-exploring/models/losses/dsd_loss.py", line 108, in forward
    "gen_im_lr": self.D.adaptKernel(gen_im, kernel),
  File "/work/deblur/blur-kernel-space-exploring/models/kernel_encoding/kernel_wizard.py", line 157, in adaptKernel
    out = self.adapter(x_sharp, kernel)
  File "/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "//work/deblur/blur-kernel-space-exploring/models/kernel_encoding/kernel_wizard.py", line 99, in forward
    return self.model(x, k)
  File "/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/deblur/blur-kernel-space-exploring/models/backbones/unet_parts.py", line 103, in forward
    return self.up(self.submodule(self.down(x), noise))
  File "/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/deblur/blur-kernel-space-exploring/models/backbones/unet_parts.py", line 109, in forward
    return torch.cat((self.up(self.submodule(self.down(x), noise)), x), dim=1)
  File "/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/deblur/blur-kernel-space-exploring/models/backbones/unet_parts.py", line 109, in forward
    return torch.cat((self.up(self.submodule(self.down(x), noise)), x), dim=1)
  File "/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/deblur/blur-kernel-space-exploring/models/backbones/unet_parts.py", line 109, in forward
    return torch.cat((self.up(self.submodule(self.down(x), noise)), x), dim=1)
  File "/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/deblur/blur-kernel-space-exploring/models/backbones/unet_parts.py", line 107, in forward
    return torch.cat((self.up(torch.cat((self.down(x), noise), dim=1)), x), dim=1)
RuntimeError: Sizes of tensors must match except in dimension 2. Got 2 and 4 (The offending index is 0)

Thank you for your help!

Please check this

I think you mean it to be F.

FileNotFoundError: [Errno 2] No such file or directory: 'experiments/pretrained/GOPRO_wVAE.pth'

Where is GOPRO_wVAE.pth?

Hi,

Thanks for this awesome work. It looks like the pretrained models don't seem to be available to download? I tried to download all 4 of them (e.g. http://public.vinai.io/models/blur-kernel-space-exploring/REDS_woVAE.pth) They don't seem to be downloadable... Is it possible to provide a public google drive link?

Thank you so much,

Missing pth pretrained models

http://public.vinai.io/models/blur-kernel-space-exploring/REDS_woVAE.pth
http://public.vinai.io/models/blur-kernel-space-exploring/GOPRO_woVAE.pth
http://public.vinai.io/models/blur-kernel-space-exploring/GOPRO_wVAE.pth
http://public.vinai.io/models/blur-kernel-space-exploring/mix_woVAE.pth

All the above links do not work.

Any other alternative sources for downloading?

To transfer the motion blur between the image pair (x, y) to a new image xˆ, we can simply compute: yˆ := F(ˆx, G(x, y)). however, when i set the x^ same as the x, the output y^ is largely different from y. It seems that the extracted kernel is not that accurate。

Hello
How are you?
Thanks for contributing to this project.
I checked that the ONLY "GOPRO wVAE" model can be used to generate blurry image.
But I know that the GOPRO dataset contains ONLY motion blur.
Therefore, I think that your trained model "GOPRO wVAE" can generate ONLY motion blurry images.
I want to generate out-of-focus blurry image too.
Can your trained models be used to generate out-of-focus blurry image?
If not so, can we train such model by your method?
If we train such model, can we train on a mixed dataset consisting of both motion deblurring dataset and defocus deblurring dataset?

Hi!
First of all congratulations for the work.

In the code, the output of the postprocessblock is added to the input sharp image in order to generate the blur image.
In the paper i do not find any place that says explicitly that the output is a residual to be added to the residual in order to create the blur image. is it a bug? should be added? Or F computes directly the blur image?

Thanks!

Hi. I'm trying to deblur face01.png image of blur_faces. While I was running generic_deblur.py with default settings with pretrained weights of GOPRO_woVAE.pth and the results were unsatisfactory. I checked that the generate_blur.py made satisfactory results of blur generating process. For prerequisites, I'm using the versons below.

• Python = 3.9.5
• Pytorch = 1.9.0
• CUDA = 10.1

I tried to use Pytorch version of 1.4.0 but version error occured, so I used 1.9.0.
I want to get results in the paper for face01.png image, and if there is anything I should tune or change, please let me know.
Thank you for your help.

When I run generic_deblur.py， I got CUDA memory Error. Is there some way to use less memory to debulr?
Thank you.

generate_blur.py:30: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
opt = yaml.load(f)["KernelWizard"]
Sample #0/10
Traceback (most recent call last):
File "generate_blur.py", line 53, in
main()
File "generate_blur.py", line 45, in main
LQ_tensor = model.adaptKernel(HQ_tensor, kernel)
File "C:\Users\Xun.Huang\blur-kernel-space-exploring\models\kernel_encoding\kernel_wizard.py", line 157, in adaptKernel
out = self.adapter(x_sharp, kernel)
File "C:\Users\Xun.Huang\Anaconda3\envs\pytorch-env\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\Xun.Huang\blur-kernel-space-exploring\models\kernel_encoding\kernel_wizard.py", line 99, in forward
return self.model(x, k)
File "C:\Users\Xun.Huang\Anaconda3\envs\pytorch-env\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\Xun.Huang\blur-kernel-space-exploring\models\backbones\unet_parts.py", line 103, in forward
return self.up(self.submodule(self.down(x), noise))
File "C:\Users\Xun.Huang\Anaconda3\envs\pytorch-env\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\Xun.Huang\blur-kernel-space-exploring\models\backbones\unet_parts.py", line 109, in forward
return torch.cat((self.up(self.submodule(self.down(x), noise)), x), dim=1)
File "C:\Users\Xun.Huang\Anaconda3\envs\pytorch-env\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\Xun.Huang\blur-kernel-space-exploring\models\backbones\unet_parts.py", line 109, in forward
return torch.cat((self.up(self.submodule(self.down(x), noise)), x), dim=1)
File "C:\Users\Xun.Huang\Anaconda3\envs\pytorch-env\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\Xun.Huang\blur-kernel-space-exploring\models\backbones\unet_parts.py", line 109, in forward
return torch.cat((self.up(self.submodule(self.down(x), noise)), x), dim=1)
File "C:\Users\Xun.Huang\Anaconda3\envs\pytorch-env\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\Xun.Huang\blur-kernel-space-exploring\models\backbones\unet_parts.py", line 107, in forward
return torch.cat((self.up(torch.cat((self.down(x), noise), dim=1)), x), dim=1)
RuntimeError: torch.cat(): Sizes of tensors must match except in dimension 1. Got 4 and 2 in dimension 2 (The offending index is 1)

the pre-trained model file is GOPRO_wVAE . anyone can help me? Should i change the dim=1 to dim=2?

When I give input a pair of sharp-blur images with the same size of 720x1280, and I try to transfer motion blur to the target image also with the same size of 720x1280. I modified the code so that instead of taking a random sample, it takes the entire size of the target image, but the return result is as follows. Can you help me answer this, thank you