Can the input image only be square pixels? If not, how to set parameters during testing

如题， 是LGPL,BSD，还是其他的？多谢

Such as this line, token_embed will affect the choice of linear, linear_concat and conv.

i have tried in val stage, but the training samples are not square, so i cropped them to (256, 256), but it influence the results.

When will we see the source code for this work?

为什么Input Projection里要把x的维度变成B H*W C？不是很理解这里

Maybe you can padding the image first, otherwise invalid information will be fused in the large scale token:

def expand2square(timg, factor=128):
	# padding first
	_, _, h, w = timg.size()
	mod_pad_h = (factor - h % factor) % factor
	mod_pad_w = (factor - w % factor) % factor
	timg = F.pad(timg, (0, mod_pad_w, 0, mod_pad_h), 'reflect')

	_, _, h, w = timg.size()
	X = int(math.ceil(max(h,w)/float(factor))*factor)

	img = torch.zeros(1,3,X,X).type_as(timg) 	# 3,h,w
	mask = torch.ones(1,1,X,X).type_as(timg)	# for -inf

	img[:, :, :h,:w] = timg
	mask[:, :, :h,:w].fill_(0.0)
	
	return img, mask

This work is interesting, when will the code release?

Hi, I am using the Rain200H dataset with a resolution of 481321 cropped to 128128 input size, but I am having the following problem.

感谢您在图像恢复领域出色的工作，您提出的Uformer对于后来者的工作具有非常深远的借鉴意义。
关于图像去模糊的code您还没有公布，请问您能公开图像去模糊的code吗？
再次感谢！

Same the title, can you please share how to edit the source code to train deraining task using YCbCr color space as the paper hints? Thank you

Sir/madam, your model is doing great and results are crisp, can we use it for commercial purpose

This is my testing code:

import torch
import math
from model import Uformer

def expand2square(timg, factor=16.0):
_, _, h, w = timg.size()

X = int(math.ceil(max(h, w) / float(factor)) * factor)

img = torch.zeros(1, 3, X, X).type_as(timg)  # 3, h,w
mask = torch.zeros(1, 1, X, X).type_as(timg)

# print(img.size(),mask.size())
# print((X - h)//2, (X - h)//2+h, (X - w)//2, (X - w)//2+w)
img[:, :, ((X - h) // 2):((X - h) // 2 + h), ((X - w) // 2):((X - w) // 2 + w)] = timg
mask[:, :, ((X - h) // 2):((X - h) // 2 + h), ((X - w) // 2):((X - w) // 2 + w)].fill_(1.0)

return img, mask

x = torch.randn(1, 3, 128, 720)

rgb_noisy, mask = expand2square(x, factor=128)

print(rgb_noisy.size())
print(mask.size())
out = model(rgb_noisy, (1 - mask))
print(out.size())

I meet the following problem:

torch.Size([1, 3, 768, 768])
torch.Size([1, 1, 768, 768])
Traceback (most recent call last):
File "/Users/Uformer-main/my_test.py", line 69, in
out = model(rgb_noisy, (1 - mask))
File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/Users/Uformer-main/model.py", line 1336, in forward
conv0 = self.encoderlayer_0(y,mask=mask)
File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/Users/Uformer-main/model.py", line 1044, in forward
x = blk(x,mask)
File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/Users/Uformer-main/model.py", line 695, in forward
attn_mask = attn_mask or shift_attn_mask
RuntimeError: bool value of Tensor with more than one value is ambiguous

hello,Where are Modulators in your code？is it qk_bias?

hi, thanks for your meaningful work
previous work about Transformer in vision hava a common opinion
which transformer needs huge dataset to feed if you want its performence great

in this work, you just train the network in SIDD patches, which nearly about 9w patches , but other works train their Transformer in nealy 100w.

so, can you explain this reason?
or can i say Transformer actually does not need too many data to feed?

Hi your work is very inspiring!

I didn't find in your paper on how you apply Uformer during inference. For example, on SIDD, the training patches are 128x128, and evaluation patches are 256x256. Were you directly applying your network on the whole 256x256 patch, or in a sliding window form? In other words, does Uformer supports arbitrary input resolution?

Hello.

I looked at the code and read the paper, and windowed attention(Window-based Multi-head Self-Attention (W-MSA)) was first proposed in "image restoration using swin transformer."

However, you did not mention the exact difference in your paper.

What is difference?

hi there, thanks for your job for offerring a script "test_in_any_resolution.py"
but, in this script, a image with random size has been processed in expand2square function,
but such size cant feed into the Uformer model.

so i wonder if this network cant process the size isn't (256,256)?
if i want to denoise the image with random size, i have to resize the size of image to (256,256)?
thanks!

Thank you for your work. If the input size of the network is 256 x 256, should I increase one encoder and decoder layer to keep the bottleneck layer's feature size is 8 (equal to window-size)? Or keep the layer num and the bottleneck's is 16, which one performs better, please? The former will bring more computation cost. In another way, is it a trick that the feature size in bottleneck is equal to the window size, please?

Thanks for sharing your great work! I meet some problems when submitting my result to SIDD benchmark. It told me that my submission file (960 MB) was over the limitation (900 MB). I used the SIDD Benchmark Data (Noisy sRGB data)
.
To generate .mat files, I used python to change image into .mat (the same way as MPRNet)...

Hi, I found that the value of SSIM in your paper (0.970) on SIDD is different and significantly higher than in paperswithcode(0.959). Your test code uses the structural_similarity method to calculate SSIM, using this method I can also get the same result as in your paper, but is this result comparable with other papers?

I can't find where the "Modulator" in the paper is implemented in model.py code.
Could you please help find it?

hi there! Thanks for your wonderful work!
This is amazing to train Transformer in just SIDD(not a huge dataset)
But when i read paper, I notice that Window-based self-attention can ruduce computation al cost.

if feature map X(C, W, H)
global self-attention cost : O(H2W2C)
Window-based self-attention cost : O(HW/M2M4C) = O(M2HWC)

i have no idea why global is O(H2W2C) and why window-based is O(HW/M2M4C)
could you please explain detials of this fomulation?
Expect your reply!

Hi, @ZhendongWang6 Thanks a lot for your wonderful work in Uformer. I have noticed that the new version of Uformer have updated in arxiv about one month ago. The modulator seems to improve the performance a lot. So I wonder when will the new version of code be released?

Again, thanks a lot for your excellent job :)

Good Job !
Where can we get the pertrained model?

Thanks !

Hi, you use SIDD-Medium sRGB data, right? but there are mirror 1 and mirror 2, so which mirror do you use? and what's the difference of mirror 1 and 2?

您好，我看到您的arxiv论文上是有多尺度调制的，为什么这个代码好像没有这个模块了，而且这个代码里附的图也有比较大的变化。希望能得到您的回答，非常感谢！

I trained the model you provided on the GoPro dataset.
I trained 3000 Epoch with CharbonnierLoss and Edge loss as MPRNet code and I also use same scheduler as MPRNet code

but I got the PSNR 31.1db on GoPro test dataset

Is this happens because of the absence of modulator?
or is there any other problem?

Hi your code curently dont support non-square entry, even if I change img_size for img_size_H and img_size_W, there is still an error because the input is not a square. This is a very big limitation of your code right now..

Kind regards,
Sébastien de Blois

test_in_any_resolution.py has an error and It is the full stack of error message.

 `Traceback (most recent call last):
  File "/content//Uformer/test_in_any_resolution.py", line 104, in <module>
    rgb_restored = model_restoration(rgb_noisy, 1 - mask)
  
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/parallel/data_parallel.py", line 159, in forward
    return self.module(*inputs[0], **kwargs[0])
  
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)

TypeError: forward() takes 2 positional arguments but 3 were given`

Actually, I was trying with my custom data but Its shape is the same as sidd data so I can't understand why.

Hi,

Thanks for your wonderful work of Uformer!
I'm wondering is there a plan to release the pretrain weight train on GOPRO?
Thank you!

When I trained on other datasets, the block effect occurred in the results. Have you ever been in a similar situation? How did you solve it?
Looking forward to your reply. Thank you

Uformer的forward函数中的mask是指待修复图像的掩膜么？

I used the same package as your code. (from ptflops import get_model_complexity_info)

And got 2.51 GMac, 5.15 M for Uformer16, 9.98 GMac 20.47 M for Uformer32.

It seems that I need to define some ops in this model. Can you provide a solution or relative code on computing Flops and Params?

Thanks!

作者您好，我看代码，是将数据都划分成patches再放到模型中，测试也是一样，但是这样的话该如何得到完整的图像呢？
我尝试过将各个patches拼接起来，但是这样的话会有明显的划分痕迹，所以想请教一下作者论文中的结果如何得到的呢？

hi friend:
Uformer is interesting. Paper reports that you train Uformer_16 for 250 epochs with batch size 32 to get 39.66 PNSR in SIDD. So how many iters in training phase？What is PSNR when 40 epochs are trained？ I just want to reproduce this result in a short time

do you think retrain your model on dehazing work andk it will be useful？

In model.py at 475 line(flops of LeFF),

    def flops(self, H, W):
        flops = 0
        # fc1
        flops += H*W*self.dim*self.hidden_dim 
        # dwconv
        flops += H*W*self.hidden_dim*3*3
        # fc2
        flops += H*W*self.hidden_dim*self.dim
        print("LeFF:{%.2f}"%(flops/1e9))
        return flops

flops of dwconv is calculated by
flops += H*W*self.hidden_dim*3*3
would is it like to be
flops += H*W*self.hidden_dim*self.hidden_dim*3*3
as the flops of conv cal as c_ink_hk_wc_outH*W ?

Hi,

How can we train the uformer for deblurring tasks?

I mean train the uformer on GoPro dataset

Thanks

您好，请问可以在模型的训练和测试中使用.mat 或者.dng格式的图片吗？

hello! thanks for your devotion.
I train the Uformer using the SIDD on 2 V100 as you suggestion.
i trained nearly 69 epoches and stoped it. i got a weight file.
i valid it and it perform well.
but when i want to fintune on 2 V100, i add --resume command line.

in train.py line 169: loss_scaler(loss, optimizer, parameters=model_restoration.parameters())
i raise a RuntimeError:Expeted all tensors to be on the same device, but found at least two devices, cude:0 and cpu

i dont know how to solve this matter, have you met this problem?
thx!

The provided link needs permission. https://drive.google.com/drive/folders/1LzYL4GYH5U40rae_4jvy8kzOtsiMeXFn
Is that the 'SIDD Benchmark Data'?

Hi, thanks for public your awesome works,
Currently, the provided denoising model is very nice, But it does not work well with rain images. May I ask if you will public other checkpoints of like remove rain, deblur? Or please give some suggestions to train them using your model. Thanks

Good Jobs ！！！
期待代码

I used the tutorial and got patched inputs but is there any tutorial to get unified results or should I make one of them?