xpixelgroup / basicsr Goto Github PK

I want to test the SR and SRGAN model. Now, I find a question :
Traceback (most recent call last):
File "test.py", line 49, in
for data in test_loader:
…
…
File "/opt/BasicSR-master/codes/data/util.py", line 262, in calculate_weights_indices
if not math.isclose(weights_zero_tmp[0], 0.0, rel_tol=1e-6):
TypeError: a float is required

I tried to tranform the weights_zero_tmp[0] to floattensor, but it still don't work.

weights_zero_tmp_1 = torch.sum((weights == 0), 0)
print(type(weights_zero_tmp_1)) #torch.ByteTensor
weights_zero_tmp = weights_zero_tmp_1.float()
print(type(weights_zero_tmp)) torch.FloatTensor
print(type(weights_zero_tmp[0])) torch.FloatTensor
print(weights_zero_tmp[0]) #120 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 120[torch.FloatTensor of size 18]#
if not math.isclose(weights_zero_tmp[0], 0.0, rel_tol=1e-6):
indices = indices.narrow(1, 1, P - 2)
weights = weights.narrow(1, 1, P - 2)
if not math.isclose(weights_zero_tmp[-1], 0.0, rel_tol=1e-6):
indices = indices.narrow(1, 0, P - 2)
weights = weights.narrow(1, 0, P - 2)

When I try to use urban100 with SFTGAN, I get the following error:

Traceback (most recent call last):
File "train.py", line 157, in
main()
File "train.py", line 71, in main
for i, train_data in enumerate(train_loader):
File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 286, in next
return self._process_next_batch(batch)
File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 307, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
FileNotFoundError: Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 57, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 57, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/onurctirtir/Desktop/BasicSR/codes/data/LRHR_seg_bg_dataset.py", line 58, in getitem
seg = torch.load(HR_path.replace('/img/', '/bicseg/').replace('.png', '.pth'))
File "/usr/local/lib/python3.5/dist-packages/torch/serialization.py", line 301, in load
f = open(f, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/home/onurctirtir/Desktop/datasets/urban100/img_002.pth'

Probably something is going wrong with me. A detailed description for "preparing dataset" would be super.

line 28 in codes/test_seg.py
load_path = '../experiments/pretrained_models/OutdoorSceneSeg_bic.pth'
Should we train a segmentation model by ourselves？

Thanks for your excellent repository! But I wonder if there is any difference between the OpenCV build-in Bicubic interpolation and the Bicubic algorithm you implemented in codes/data/util.py.

When I trained my own model using my database, it worked well. But unfortunately if I ran the code in windows anaconda，it showed the error like this:

It maybe some problem about pytorch multiprocessing in windows.
If someone knows how to solve it,pls help! Thx~

I don't know if opening an issue for this is appropriate or not but I stuck around at some point:

1. In SFTGAN.json, what is expected for "datasets:dataroot_HR_bg:" ? What 'bg' corresponds for ?
2. If I'm not wrong, I should put HR images under 'img/' at some directory and '.pth' files under 'bicseg/' being in the same directory with 'img/'

Thanks in advance

Hi,

I was wondering is it possible with BasicSR/ESRGAN to create real-time 2x or 4x higher resolution from a video stream of 512x448 pixels with let's say 20 or 30 FPS?

Real-time, I mean a stream of input data, which creates a stream of output data (eg. like a movie as input).

Kind regards,
Melroy

I want to use DIV2K800 dataset to train SFT-GAN, but I don't know which folder should I put DIV2K800 segmentation .pth files? And does the OutdoorScene testing dataset require my manual classification?

When I use the your model to train the sr model. I found a problem, the l_pix is stable at around 0.02, is this normal? I have not changed any parameters, it is the same as your model.

the default activation_type you use in generator is 'leakyrelu'. have you tried doing as DCGAN paper suggests? they claim that one should use relu for all layers in generator except for the output where the tanh should be used.

when i run "train.py",it shows that "ConnectionRefusedError: [Errno 111] Connection refused"
line 156:main();then i add mp.set_start_method,but it not working

The links of OST cannot be opened.

When I test the model on the Set5 dataset with the pretrained weights in RRDB_PSNR_x4.pth, I get a psnr score of 15.0664 as opposed to 25+ reported in the paper. Do you know what the issue might be? I haven't modified the training code and ran the image preprocessing scripts as reported in the paper.

The only difference is that I used octave instead of matlab to run the downsampling script. The script produced uint16 images, which I converted to uint8 using the following code:

    img = cv2.imread(v, cv2.IMREAD_UNCHANGED)
    img = img.astype(np.float32)
    img = (img * 255.0 / 65535.0).round()
    img = img.astype(np.uint8)
    cv2.imwrite(v, img)

I used matlab to reduce the resolution of the image, but why the test results obtained with the pre-training model are two points lower than the results given.

what is the dataroot_HR_bg in train_sftgan.json?

I'm just getting into it and already got a problem with testing the pre-trained Esrgan model.

When running it I get the following error message which seems to be a windows related Pytorch issue:

File "C:\Users\Steven\Anaconda3\lib\multiprocessing\spawn.py", line 136, in 
_check_not_importing_main
is not going to be frozen to produce an executable.''')
  RuntimeError:
    An attempt has been made to start a new process before the
    current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Any idea how to fix it?

Hi,

If I need scale x3 not x4, should I modify the model and train it from scratch? Where do I need to modify..?

Thank you

I'm new to this. I'm curious that when I train the model, I got G.pth and D.pth. While in the option files, e.g., test_esrgan.json, you only provide option for "pretrain_model_G", then how about model_D? What if I want to use pretrained model_D?

The following error pops up：

from . import block as B
ImportError: cannot import name 'block'

Thanks a lot!

Hi,

Just for clarification, why is the data randomly scaled in addition to the x4 upscale in SFTGAN? Is it just for purposes of data augmentation?

Also, is there a simple way of pre-generating the low-res images using mat-lab? The way the logic is implemented, (since it also randomly samples images from the DIV2k dataset), it seems impossible to simply put the LR images in a directory and complete the "dataroot_LR": data field in train_sftgan.json. Is that correct? I can understand ERGAN implementation but SFTGAN implementation is still a bit confusing to me.

Thanks

If I change the architecture of the model, can I use the pretrained model RRDB_PSNR_x4.pth?

Traceback (most recent call last):
File "C:/BasicSR/codes/train.py", line 158, in
main()
File "C:/BasicSR/codes/train.py", line 22, in main
opt = option.parse(parser.parse_args().opt, is_train=True)
File "C:\BasicSR\codes\options\options.py", line 18, in parse
opt = json.loads(json_str, object_pairs_hook=OrderedDict)
File "C:\Users\Mixtresh\AppData\Local\Programs\Python\Python36\lib\json__init
__.py", line 367, in loads
return cls(**kw).decode(s)
File "C:\Users\Mixtresh\AppData\Local\Programs\Python\Python36\lib\json\decode
r.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Mixtresh\AppData\Local\Programs\Python\Python36\lib\json\decode
r.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

i run train on windows 8.1

Hello,

I encounter the error UnboundLocalError: local variable 'val_loader' referenced before assignment when trying to train SR. The execution command is python train.py -opt options/train/train_sr.json.

Do any one have an idea why this happens?

The full output and error is as follows and is encountered after iteration 8 in the validation phase:

Path already exists. Rename it to [/home/.../.../basicsr-xinntao/experiments/debug_001_RRDB_PSNR_x4_DIV2K_archived_181127-104642]
Random Seed:  0
Dataset [LRHRDataset - Sobolt] is created.
Number of train images: 9,687, iters: 606
Total epoches needed: 1651 for iters 1,000,000
initialization method [kaiming]
---------- Model initialized ------------------
Number of parameters in G: 16,697,987
-----------------------------------------
Model [SRModel] is created.
---------- Start training -------------
/opt/anaconda3/lib/python3.6/site-packages/torch/nn/modules/upsampling.py:122: UserWarning: nn.Upsampling is deprecated. Use nn.functional.interpolate instead.
  warnings.warn("nn.Upsampling is deprecated. Use nn.functional.interpolate instead.")
<epoch:  0, iter:       2, time:1.31, lr:2.0e-04> l_pix: 4.21e-01
<epoch:  0, iter:       4, time:1.24, lr:2.0e-04> l_pix: 3.85e-01
<epoch:  0, iter:       6, time:1.24, lr:2.0e-04> l_pix: 3.41e-01 
<epoch:  0, iter:       8, time:1.25, lr:2.0e-04> l_pix: 2.01e-01 
Saving the model at the end of iter 8.
---------- validation -------------
Traceback (most recent call last):
  File "train.py", line 156, in <module>
    main()
  File "train.py", line 106, in main
    for val_data in val_loader:
UnboundLocalError: local variable 'val_loader' referenced before assignment

I am using 4 GPUs, a batch_size of 16 with 8 n_workers and an HR_size of 512.

Any help is appreciated.

Hi. Have you ever noticed that the detail or texture always changes frequently during training GAN? If yes, why???

Sorry, I didn't find the options folder as well as config.json. Could you please check again ? Thank you

I get the below issue while training with both PSNR and ESRGAN pre-trained models.

  RuntimeError: Error(s) in loading state_dict for RRDBNet:
 Missing key(s) in state_dict: "model.1.sub.16.bias", "model.1.sub.16.weight".

Is it because the pre-trained model is now not compatible with RRDBNet model?

when I run train.py I get:

19-01-02 21:35:49.755 - INFO: Model [SRRaGANModel] is created.
19-01-02 21:35:49.773 - INFO: Start training...
Traceback (most recent call last):
  File "train.py", line 154, in <module>
    main()
  File "train.py", line 79, in main
    for i, train_data in enumerate(train_loader):
  File "C:\Users\Claes\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\data\dataloader.py", line 819, in __iter__
    return _DataLoaderIter(self)
  File "C:\Users\Claes\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\data\dataloader.py", line 560, in __init__
    w.start()
  File "C:\Users\Claes\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\Claes\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Claes\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\Claes\AppData\Local\Programs\Python\Python36\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\Claes\AppData\Local\Programs\Python\Python36\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle Environment objects

E:\projects\BasicSR\codes>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Claes\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\Claes\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

I am running windows 10 home and python 3.6.8 gtx 1060 max q

As I see, test_seg.py unsquzee's seg sensors to get 4D before saving. But train.py tries to read 3D tensors from pth files

How to write configuration for SRGAN(Not ESRGAN) testing? I always encounter the "missing key in state_dict" problem.

Hey Xinntao!
I'd like to work on your project and would like to ask you for your lmdb files so I can start faster.
Thank you very much!

I saw some parameter in train_esrgan.json，but I am not sure the meaning. So pls can you give some detail to help me?

1.Like beta1_G,beta1_D,lr_gamma in train_esrgan.json?

2.I saw in the code have the training model for wgan-gp,so can it work? Because I saw ReadMe just delete this part. So I am not sure whether you change the code or not? Did you do the experiment to use the D as wgan-gp? If you did it before, could you tell me the perceptual quality use the D as wgan-gp?

3.I am a new in Neural Networks, and never try a model in image. So I want to know at least how many image can I build the esrgan model if I use 23 DDRB? And if I don't change the architecture, can I use my pre-train mode as pretrain_model_G,then change the parameter and run the code again to get a better network?

Please help，thank you so much...

Is it possible to modify the code so the normalized pixels are in range -1..1?
I started to do it myself, but then stopped at

def rgb2ycbcr(img, only_y=True):
    '''same as matlab rgb2ycbcr
    only_y: only return Y channel
    Input:
        uint8, [0, 255]
        float, [0, 1]
    '''

won't it ruin the logic of the transformations?

How did you get the ESRGAN pre training model?

Hello, how much time do the training process spend? I find it slowly...

Hi,

I've been reading your code and wonder if the detach function on NetF is necessary.
In SRRaGAN_model.py, line 128: real_fea = self.netF(self.var_H).detach(), since we don't need to update the NetF, is that OK to remove .detach() function?
If no, then why the fake_fea = self.netF(self.fake_H) doesn't have the detach function?

[Update]
Sorry but I think I understood your code. This line is for updating the NetG then we don't need to detach the self.netF(self.fake_H) (because self.fake_H is the result of NetG). Instead, we do self.netF(self.var_H).detach() since it is not necessary.
Please close this topic if my understanding is correct.

Are following statements correct?

• sft_net_ini.pth is a model that was trained for L1 loss only on DIV2K (+probability maps for each of the 800 DIV2K images)
• ImageNet/Flickr pictures were not used to training sft_net_ini.pth or any SFTGAN model.

Tried to create my own lmdb file using the create_lmdb.py file.
I found that the script only works with different filename cause it uses that filename (basename) as key.

My path is somewhat like this:

/home/Dataset/SISR/Class_A/XXXX.png
/home/Dataset/SISR/Class_B/XXXX.png
/home/Dataset/SISR/Class_C/XXXX.png

Filenames are the same but the content is different. The create_lmdb.py will create only 1 key (it means the size of dataset will be 1 instead of 3 files)

The only solution I have right now is to rename all image filenames so that they do not duplicate. Then run the code to generate the lmdb file. But it seems to be not the convenient way.
Do you have any better ideas? Thanks

[UPDATED] My solution is to add the iterator i with basename so that they will have a unique name.
I changed the line #33 in the create_lmdb.py to:
base_name = str(i) + "_" + os.path.splitext(os.path.basename(v))[0]

I use your weight and ESRGAN model to test. I found the PSNR and SSIM is worse than that in the paper. Set 5 is 28.5438 PSNR, 0.8313 SSIM.

I'm confused about the category loss "l_d_cls_fake = self.cri_ce(cls_d_fake, self.var_cat)", which seems the same behaviour as "l_g_cls = self.l_gan_w * self.cri_ce(cls_g_fake, self.var_cat)" in generator and violation of the nature of discriminator. Shouldn't it be "l_d_cls_fake = self.cri_ce(cls_d_fake, 'background')" or something? Thank you in advance.

I noticed that you have changed the project wiki,could you please show the old project wiki about the data prepare and the options,thank you very much!!

Could you post the training set for segment model? It seems apart from COCO and ADE, you have combined some addtional images. And also, could you provide training details like hyper-parameters? I see Xiaoxiao's code but it looks like based on caffe. Since we have seg_arch.py, I'm not sure whether it is possible to train the segment model in Pytorch? Thanks in advance!

Hi, thanks for your novel work, I have some questions.

In the Relativistic GAN the sigmoid function is used little differently from standard GAN. Specifically, I see in the paper it says the Sigmoid function is added after C(X_r) - C(X_f). But I haven't found the Sigmoid function in your code. Could you please help me to find the location and usage of it?

@xinntao Hello, I found that you use the skimage.measure to compute SSIM index. But according to the offical API, this compare_ssim function should set gaussian_weight to True, sigma to 1.5, and use_sample_covariance to False to match the implementation of Matlab (Wang et. al.).
https://github.com/xinntao/BasicSR/blob/71635757505ef113bfba029441a4e35e59994a86/codes/utils/util.py#L90-L92

skimage.measure.compare_ssim

if I have stopped training due to some technical reasons, and I have previous versions of my model saved to the 'experiments' folder, how can I start not from the beginning again but from where I stopped the last time?

This is a beginner question.. In the tensorboard, there are D_fake, D_real, l_d_fake, l_d_real, l_g_fea, l_g_gan, l_g_pix. Could you please explain what they are and what the expected trend each of them should have?

Thank you so much.

Did you implemented some mechanisms to continue learning when there was a failure?
If so, how can i use it?

thanks

Why do you detach the fake_H in SRRaGAN_model.py? The update would not be applied to G, as you did not add the G parameter to the optimizer. Is that correct?

pred_d_real = self.netD(self.var_ref)
pred_d_fake = self.netD(self.fake_H.detach()) # detach to avoid BP to G

optimizer:
self.optimizer_D = torch.optim.Adam(self.netD.parameters(), lr=train_opt['lr_D'], \ weight_decay=wd_D, betas=(train_opt['beta1_D'], 0.999))

First time I ran the creat_lmdb, it occured that ZeroDivisionError: float division by zero.

if I change the code like that:

It can read, but fail to write. The error is the same... But!!! It produce the lmdb file even my cmd told me that it had error.So anyone help me please....

I'm just wondering that is it possible for this model to be able to support 16bit .tif image? Or I have to convert it to 8bit?