amanchadha / iseebetter Goto Github PK

I had trouble installing on a HPC with access to GPUs (#29 ) so I've now tried to install on my local laptop (without GPU support). Using python 3.8.0 I still get the same errors trying to install requirements.txt and there are a few dependencies that were not mentioned in this file, namely: Cython, torchvision and opencv-python:

modified requirements.txt:

#torch==1.3.0.post2
torch
pytorch-ssim==0.1
numpy
scikit-image
tqdm
#pytorch-ssim==0.1
#numpy==1.16.4
#scikit-image==0.15.0
#tqdm==4.37.0
Cython
torchvision
opencv-python

After successfully installing pyflow via:

cd pyflow/
python setup.py build_ext -i  # build pyflow
python demo.py                # to make sure pyflow works
cp pyflow*.so ..

I try to run the example iSeeBetterTest.py but I get the following error:

(venv) mtg@mtg-ThinkPad-P53:~/iSeeBetter$ python3 iSeeBetterTest.py 
Namespace(chop_forward=False, data_dir='./Vid4', debug=False, file_list='foliage_test.txt', future_frame=True, gpu_mode=False, gpus=0, model='weights/netG_epoch_4_1.pth', model_type='RBPN', nFrames=7, other_dataset=True, output='Results/', residual=False, seed=123, testBatchSize=1, threads=1, upscale_factor=4, upscale_only=False)
==> Loading datasets
==> Building model  RBPN
[    INFO] ------------- iSeeBetter Network Architecture -------------
[    INFO] ----------------- Generator Architecture ------------------
[    INFO] Net(
  (feat0): ConvBlock(
    (conv): Conv2d(3, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (act): PReLU(num_parameters=1)
  )
  (feat1): ConvBlock(
    (conv): Conv2d(8, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (act): PReLU(num_parameters=1)
  )
  (DBPN): Net(
    (feat1): ConvBlock(
      (conv): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (up1): UpBlock(
      (up_conv1): DeconvBlock(
        (deconv): ConvTranspose2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
      (up_conv2): ConvBlock(
        (conv): Conv2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
      (up_conv3): DeconvBlock(
        (deconv): ConvTranspose2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
    )
    (down1): DownBlock(
      (down_conv1): ConvBlock(
        (conv): Conv2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
      (down_conv2): DeconvBlock(
        (deconv): ConvTranspose2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
      (down_conv3): ConvBlock(
        (conv): Conv2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
    )
    (up2): UpBlock(
      (up_conv1): DeconvBlock(
        (deconv): ConvTranspose2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
      (up_conv2): ConvBlock(
        (conv): Conv2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
      (up_conv3): DeconvBlock(
        (deconv): ConvTranspose2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
    )
    (down2): DownBlock(
      (down_conv1): ConvBlock(
        (conv): Conv2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
      (down_conv2): DeconvBlock(
        (deconv): ConvTranspose2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
      (down_conv3): ConvBlock(
        (conv): Conv2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
    )
    (up3): UpBlock(
      (up_conv1): DeconvBlock(
        (deconv): ConvTranspose2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
      (up_conv2): ConvBlock(
        (conv): Conv2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
      (up_conv3): DeconvBlock(
        (deconv): ConvTranspose2d(64, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
        (act): PReLU(num_parameters=1)
      )
    )
    (output): ConvBlock(
      (conv): Conv2d(192, 64, kernel_size=(1, 1), stride=(1, 1))
    )
  )
  (res_feat1): Sequential(
    (0): ResnetBlock(
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (1): ResnetBlock(
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (2): ResnetBlock(
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (3): ResnetBlock(
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (4): ResnetBlock(
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (5): DeconvBlock(
      (deconv): ConvTranspose2d(256, 64, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
      (act): PReLU(num_parameters=1)
    )
  )
  (res_feat2): Sequential(
    (0): ResnetBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (1): ResnetBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (2): ResnetBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (3): ResnetBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (4): ResnetBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (5): ConvBlock(
      (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
  )
  (res_feat3): Sequential(
    (0): ResnetBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (1): ResnetBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (2): ResnetBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (3): ResnetBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (4): ResnetBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (act): PReLU(num_parameters=1)
    )
    (5): ConvBlock(
      (conv): Conv2d(64, 256, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
      (act): PReLU(num_parameters=1)
    )
  )
  (output): ConvBlock(
    (conv): Conv2d(384, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  )
)
[    INFO] Total number of parameters: 12771943
Pre-trained SR model loaded from: weights/netG_epoch_4_1.pth
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/mtg/.pyenv/versions/3.6.3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/mtg/.pyenv/versions/3.6.3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mtg/.pyenv/versions/3.6.3/lib/python3.6/multiprocessing/resource_sharer.py", line 139, in _serve
    signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG))
  File "/home/mtg/.pyenv/versions/3.6.3/lib/python3.6/signal.py", line 60, in pthread_sigmask
    sigs_set = _signal.pthread_sigmask(how, mask)
ValueError: signal number 32 out of range

Any idea what the issue is here?

I have a problem when testing. I run python3 iSeeBetterTest.py --output Result -c --data_dir Vid4 --file_list foliage_test.txt -m weights/netG_epoch_4_1.pth --upscale_only
my parameters are
(chop_forward=False, data_dir='Vid4', debug=False, file_list='foliage_test.txt', future_frame=True, gpu_mode=True, gpus=1, model='weights/netG_epoch_4_ 1.pth', model_type='RBPN', nFrames=7, other_dataset=True, output='Result', residual=False, seed=123, testBatchSize=1, threads=0, upscale_factor=4, upscale_only =True)

and finally I get an CUDA memory error..
Please someone help me. Thank you

Is there a colab notebook available for testing?

Just a heads up you forgot to put

opencv-python

in your requirements.txt so running the test script gives a missing dependency error.

Also, the Cython dependency was also needed to install the pyflow module for me, perhaps put it in requirements too.

i need a colab notebook PLZ
My Email : [email protected]

Hi there! You said you fine-tuned the pre-trained RBPN for 4 epochs. I was curious about the values of Generator and Discriminator loss and their behavior. Do they follow the SRGAN? Also, how did you decide to fine-tune it for 4 epochs? By observing losses and evaluation set scores?

I ran the command "python3 iSeeBetterTest.py" for evaluation and used netG_epoch_4_1.pth model.
Used the foliage_test.txt,calendar_test.txt and city_test.txt as the input for images .
I got the PSNR values25.16 , 21.66 , 25.69 respectively
But the paper reports 26.57, 24.13, 28.34 respectively
I also tried to calculate the PSNR values on all the images in all scenes of Vid.
The PSNR I got for foliage ,calendar ,city ,walk is 24.65 , 22.01 , 26.09 , 29.04 respectively which still doesn't match the result reported by the paper
Please may I know if I m missing something because of which i m not able to produce the same result

I installed all the requirements and then also downloaded the Vimeo dataset using the script that you provided : 'DatasetFetcher.py'

After that I tried to train with the command "python3 iSeeBetterTrain.py", but I got this error:

$ python3 iSeeBetterTrain.py
[    INFO] ==> Loading datasets
Training samples chosen: sep_trainlist.txt
Traceback (most recent call last):
  File "iSeeBetterTrain.py", line 264, in <module>
    main()
  File "iSeeBetterTrain.py", line 206, in main
    args.other_dataset, args.patch_size, args.future_frame)
  File "/home/iSeeBetter/data.py", line 17, in get_training_set
    transform=transform())
  File "/home/iSeeBetter/dataset.py", line 168, in __init__
    alist = [line.rstrip() for line in open(join(image_dir,file_list))]
FileNotFoundError: [Errno 2] No such file or directory: './vimeo_septuplet/sequences/sep_trainlist.txt'

How can I debug this?

Where i try to test the model with a diferents weights of netG_epoch_4_1.pth It shows me the following error:
Traceback (most recent call last):
File "/content/drive/MyDrive/iSeeBetter/utils.py", line 19, in loadPreTrainedModel
model.load_state_dict(state_dict)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 845, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Net:
Missing key(s) in state_dict: "feat0.conv.weight", "feat0.conv.bias", "feat0.act.weight", "feat1.conv.weight", "feat1.conv.bias", "feat1.act.weight", "DBPN.feat1.conv.weight", "DBPN.feat1.conv.bias", "DBPN.feat1.act.weight", "DBPN.up1.up_conv1.deconv.weight", "DBPN.up1.up_conv1.deconv.bias", "DBPN.up1.up_conv1.act.weight", "DBPN.up1.up_conv2.conv.weight", "DBPN.up1.up_conv2.conv.bias", "DBPN.up1.up_conv2.act.weight", "DBPN.up1.up_conv3.deconv.weight", ...

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "iSeeBetterTest.py", line 213, in
eval()
File "iSeeBetterTest.py", line 95, in eval
utils.loadPreTrainedModel(gpuMode=args.gpu_mode, model=model, modelPath=modelPath)
File "/content/drive/MyDrive/iSeeBetter/utils.py", line 32, in loadPreTrainedModel
model.load_state_dict(new_state_dict)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 845, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Net:
Missing key(s) in state_dict: "feat0.conv.weight", "feat0.conv.bias", "feat0.act.weight", "feat1.conv.weight", "feat1.conv.bias", "feat1.act.weight", "DBPN.feat1.conv.weight", "DBPN.feat1.conv.bias", "DBPN.feat1.act.weight", "DBPN.up1.up_conv1.deconv.weight", "DBPN.up1.up_conv1.deconv.bias", "DBPN.up1.up_conv1.act.weight", "DBPN.up1.up_conv2.conv.weight", "DBPN.up1.up_conv2.conv.bias", "DBPN.up1.up_conv2.act.weight", "DBPN.up1.up_conv3.deconv.weight", "DBPN.up1.up_conv3.deconv.bias", "DBPN.up1.up_conv3.act.weight", "DBPN.down1.down_conv1.conv.weight", "DBPN.down1.down_conv1.conv.bias", "DBPN.down1.down_conv1.act.weight", "DBPN.down1.down_conv2.deconv.weight", "DBPN.down1.down_conv2.deconv.bias", "DBPN.down1.down_conv2.act.weight", "DBPN.down1.down_conv3.conv.weight", ...

Hello, I'm on windows 10 with a RTX3080, and using python 3.7
I'm trying to run the test with default parameters and I have these errors after [INFO] Total number of parameters: 12771943:

 weights/netG_epoch_4_1.pth
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "H:\miniconda\envs\python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "H:\miniconda\envs\python37\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "H:\miniconda\envs\python37\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "H:\miniconda\envs\python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "H:\miniconda\envs\python37\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "H:\miniconda\envs\python37\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "H:\miniconda\envs\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "H:\git\VideoDebluring2\iSeeBetter\iSeeBetterTest.py", line 198, in <module>
    eval()
  File "H:\git\VideoDebluring2\iSeeBetter\iSeeBetterTest.py", line 87, in eval
    for batch in testing_data_loader:
  File "H:\miniconda\envs\python37\lib\site-packages\torch\utils\data\dataloader.py", line 359, in __iter__
    return self._get_iterator()
  File "H:\miniconda\envs\python37\lib\site-packages\torch\utils\data\dataloader.py", line 305, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "H:\miniconda\envs\python37\lib\site-packages\torch\utils\data\dataloader.py", line 918, in __init__
    w.start()
  File "H:\miniconda\envs\python37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "H:\miniconda\envs\python37\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "H:\miniconda\envs\python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "H:\miniconda\envs\python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "H:\miniconda\envs\python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "H:\miniconda\envs\python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

Can someone help ?

Thank in advance !

run：python iSeeBetter.py -m weights/RBPN_2X -s 2
the size of input frame is 1280720, I hope that I could get the output frame 25601440, but not.
Have I done wrong somewhere?

Hello, the idea of this paper is very inspiring to me, but there is a problem when reproducing your code, that is, the output SR image I trained are all black, the training code is unchanged, and the vimeO90K data set is also used. Have you ever encountered such a situation?

I don't understand this part of the test code，I think testing_data_loader is the test set data，only one image ,Why does batch have a lot of data，such as batch[0]、batch[1].......
code:
for batch in testing_data_loader:
#import pdb;pdb.set_trace()
input, target, neigbor, flow, bicubic = batch[0], batch[1], batch[2], batch[3], batch[4]
with torch.no_grad():
if cuda:
input = Variable(input).cuda(gpus_list[0])
bicubic = Variable(bicubic).cuda(gpus_list[0])
neigbor = [Variable(j).cuda(gpus_list[0]) for j in neigbor]
flow = [Variable(j).cuda(gpus_list[0]).float() for j in flow]
else:
input = Variable(input).to(device=device, dtype=torch.float)
bicubic = Variable(bicubic).to(device=device, dtype=torch.float)
neigbor = [Variable(j).to(device=device, dtype=torch.float) for j in neigbor]
flow = [Variable(j).to(device=device, dtype=torch.float) for j in flow]

Is there any example video showing of the temporal consistency over frames?
(before upsampling/after)
Thanks a lot 👍

Hi, I've tested your model weights/netG_epoch_4_1.pth, and I can't get the result of 26.57/0.773 on Foliage datasets. I just got Avg PSNR Predicted = 25.728. Could you tell me how to get it? Besides, the result of City is also under 26. These are my arguments:
Namespace(chop_forward=False, data_dir='./Vid4', debug=False, file_list='city_test.txt', future_frame=True, gpu_mode=False, gpus=1, model='weights/netG_epoch_4_1.pth', model_type='RBPN', nFrames=7, other_dataset=True, output='Results/', residual=False, seed=123, testBatchSize=1, threads=1, upscale_factor=4)

PS: Could you offer the augmented dataset? Thanks very much!

We have made some objective quality tests upscaling SD content to HD and the results were really impressive, much higher Netflix VMAF than ESPCN, congratulations!!

But now we are trying to upscale HD 1080 content to 4K and even with a GeForce Quadro RTX 8000 we are getting out of memory issues, so we tried using CPU mode but it apparently doesn't work.

Please help!

because of cuda 11.1

I'm running into errors with the installation of pyflow:

(venv) [moldach@beluga4 pyflow]$ python setup.py build_ext -i
running build_ext
building 'pyflow' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/scratch/moldach/iSeeBetter/venv/lib64/python3.6/site-packages/numpy/core/include -I/scratch/moldach/iSeeBetter/venv/lib64/python3.6/site-packages/numpy/core/include -I/scratch/moldach/iSeeBetter/venv/include -I/usr/include/python3.6m -c pyflow.cpp -o build/temp.linux-x86_64-3.6/pyflow.o
pyflow.cpp:29:10: fatal error: Python.h: No such file or directory
 #include "Python.h"
          ^~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1
(venv) [moldach@beluga4 pyflow]$ 

pip freeze

Actually the problem beings right-out-of-the-gate as when I created a python virtual env with version 3.6.3 and tried to run pip3 install -r requirements.txt there were errors for the specific versions listed because the distributions cannot be found (_commented out):

#torch==1.3.0.post2
torch
pytorch-ssim==0.1
#numpy==1.16.4
numpy
#scikit-image==0.15.0
scikit-image
tqdm==4.37.0

For example,

(venv) [moldach@beluga4 iSeeBetter]$ pip3 install torch==1.3.0.post2
Looking in links: /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/avx512, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/avx2, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/generic, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic
ERROR: Could not find a version that satisfies the requirement torch==1.3.0.post2
ERROR: No matching distribution found for torch==1.3.0.post2

So the environment looks like:

(venv) [moldach@beluga4 iSeeBetter]$ pip freeze
DEPRECATION: --find-links option in pip freeze is deprecated. pip 21.2 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/9069.
-f /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/avx512
-f /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/avx2
-f /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/generic
-f /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic
alabaster==0.7.12
appdirs==1.4.4
astor==0.8.1
attrs==20.3.0
autoflake==1.4
Babel==2.9.0
bandit==1.7.0
black==20.8b1
certifi==2020.12.5
chardet==4.0.0
click==7.1.2
coverage==5.5
cycler==0.10.0
Cython==0.29.22
darglint==1.7.0
dataclasses==0.8
distlib==0.3.1
doc8==0.8.1
docutils==0.16
eradicate==1.0
ez-setup==0.9
filelock==3.0.12
flake8==3.9.0
flake8-2020==1.6.0
flake8-bandit==2.1.0
flake8-broken-line==0.2.1
flake8-bugbear==19.8.0
flake8-builtins==1.5.3
flake8-commas==2.0.0
flake8-comprehensions==3.4.0
flake8-debugger==3.2.0
flake8-docstrings==1.6.0
flake8-eradicate==0.3.0
flake8-fixme==1.1.1
flake8-isort==3.0.1
flake8-logging-format==0.6.0
flake8-mutable==1.2.0
flake8-polyfill==1.0.2
flake8-quotes==2.1.2
flake8-rst-docstrings==0.0.12
flake8-string-format==0.2.3
flake8-variables-names==0.0.3
gitdb==4.0.5
GitPython==3.1.14
idna==2.10
imagesize==1.2.0
importlib-metadata==3.7.3
importlib-resources==5.1.2
iniconfig==1.1.1
isort==4.3.21
Jinja2==2.11.3
kiwisolver==1.3.1
libpython==0.2
MarkupSafe==1.1.1
matplotlib==3.3.2
mccabe==0.6.1
mypy-extensions==0.4.3
numpy==1.19.1
packaging==20.9
pathspec==0.8.1
pbr==5.5.1
pep8-naming==0.9.1
Pillow==7.2.0
pluggy==0.13.1
py==1.10.0
pycodestyle==2.7.0
pydocstyle==5.1.1
pyflakes==2.3.0
Pygments==2.8.1
pyparsing==2.4.7
pytest==6.2.2
pytest-cov==2.11.1
python-dateutil==2.8.1
python-dev-tools==2020.9.10
pytz==2021.1
pyupgrade==2.11.0
PyYAML==5.3.1
regex==2020.11.13
requests==2.25.1
restructuredtext-lint==1.3.2
six==1.15.0
smmap==3.0.5
snowballstemmer==2.1.0
Sphinx==2.4.4
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
stevedore==3.3.0
testfixtures==6.17.1
tokenize-rt==4.1.0
toml==0.10.2
tox==3.23.0
tox-travis==0.12
typed-ast==1.4.1
typing-extensions==3.7.4.3
urllib3==1.26.4
virtualenv==20.4.3
wemake-python-styleguide==0.14.1
zipp==3.4.1

When using netG_epoch_4_1_APITLoss.pth the output is less smooth than the L1 loss model as expected, but there's a faint mosaic pattern all over everything. Did you notice this in your research or is there a problem with my setup?

From what I understand the model(s) first downscale the source frames to LR and then upscale them again, trying to reach as much similarity to the original HR images...
What settings in e.g. the iSeeBetterTest.py file to try up-scaling without pictures being downscaled first?

Thank you so much for your responses.
This work is amazing!

Hi, your work is pretty impressive. I may have found a little bug. I am testing ISB in Colab in GPU mode and when I run:

!python3 iSeeBetterTest.py -c --threads 8

I get a run time error with the following output:

`Namespace(chop_forward=False, data_dir='./Vid4', debug=False, file_list='foliage_test.txt', future_frame=True, gpu_mode=True, gpus=1, model='weights/netG_epoch_4_1.pth', model_type='RBPN', nFrames=7, other_dataset=True, output='Results/', residual=False, seed=123, testBatchSize=1, threads=8, upscale_factor=4)
Using GPU mode
==> Loading datasets
==> Building model RBPN
[ INFO] ------------- iSeeBetter Network Architecture -------------
[ INFO] ----------------- Generator Architecture ------------------
[ INFO] DataParallel(
(module): Net(

..............XXXX................OMITTED................XXXX...............

(output): ConvBlock(
  (conv): Conv2d(384, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
)

)
)
[ INFO] Total number of parameters: 12771943
Traceback (most recent call last):
File "/content/drive/My Drive/isb/iSeeBetter/utils.py", line 19, in loadPreTrainedModel
model.load_state_dict(state_dict)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 839, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DataParallel:
Missing key(s) in state_dict: "module.feat0.conv.weight", "module.feat0.conv.bias", "module.feat0.act.weight", "module.feat1.conv.weight", "module.feat1.conv.bias",
..............XXXX................OMITTED................XXXX............... "module.output.conv.weight", "module.output.conv.bias".
Unexpected key(s) in state_dict: "feat0.conv.weight", "feat0.conv.bias", "feat0.act.weight", "feat1.conv.weight", "feat1.conv.bias", "feat1.act.weight",
..............XXXX................OMITTED................XXXX............... "res_feat3.5.act.weight", "output.conv.weight", "output.conv.bias".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "iSeeBetterTest.py", line 194, in
eval()
File "iSeeBetterTest.py", line 79, in eval
utils.loadPreTrainedModel(gpuMode=args.gpu_mode, model=model, modelPath=modelPath)
File "/content/drive/My Drive/isb/iSeeBetter/utils.py", line 35, in loadPreTrainedModel
model.load_state_dict(new_state_dict)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 839, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DataParallel:
Missing key(s) in state_dict: "module.feat0.conv.weight", "module.feat0.conv.bias", "module.feat0.act.weight", "module.feat1.conv.weight", "module.feat1.conv.bias",
..............XXXX................OMITTED................XXXX...............
"module.output.conv.weight", "module.output.conv.bias".
Unexpected key(s) in state_dict: "onv.weight", "onv.bias", "ct.weight", "at1.conv.weight", "at1.conv.bias", "at1.act.weight", "1.up_conv1.deconv.weight", "1.up_conv1.deconv.bias",
..............XXXX................OMITTED................XXXX...............
"conv.weight", "conv.bias".
`

Issue seems to be that the catch block in

loadPreTrainedModel(gpuMode, model, modelPath)

function in iSeeBetter/utils.py is removing "module." from the keys in (new_)state_dict, whereas the pipline in that particular state seems to need "module." to be prepended to the front of each key instead. Changing these two lines

name = k[len('module.'):] # remove module.
new_state_dict[name] = v

to

new_state_dict["module." + k] = v

seems to fix the problem for me on Colab, however I am not an expert so I am not sure if this causes any other problems. Hope it helps someone.

Full Output from !python3 iSeeBetterTest.py -c --threads 8 attached below.
out.txt

python3 iSeeBetterTest.py中如何设置参数，能够直接输入视频进行超分？

Hi there, can you please check again your model code? I run the test file with your pretrained models and all default args (with and without --upscale_only flag), the output result image is always the same as the input's. It evenly looks blurrier than the input itself, as upscaled and downscaled again I think? I also tried running with different args but it seems the problem is in your model's output. Also tried with different random images, the output is the same as well.

Hey there! I keep getting the following error:
module 'pyflow' has no attribute 'coarse2fine_flow'
when trying to run python3.7 iSeeBetterTest.py

I am using python 3.7.9 with all the corresponding packages. My torch install however is version 1.7.1 as the version in requirements.txt does not seem to exist? Anyone that can help me?

Does the provided pretrained model only have 4 epochs?

Hello
I have noticed that, when calculated the G or the D loss, you have divided it by the data length in your code, is there any usage of this operation? Waiting for your answer, Thanks a lot!

Hi, your work is very inspiring! I got errors when i run the train and then the test file. I don't know what I'm doing wrong. I would be grateful if anyone can help.

(venv) amperiad@cuda-pc:~/iSeeBetter-master$ python3 iSeeBetterTrain.py
[ INFO] ==> Loading datasets
Training samples chosen: foliage_test.txt
[ INFO] # of Generator parameters: 12771943
[ INFO] # of Discriminator parameters: 5215425
[ INFO] # of CUDA devices detected: 1
[ INFO] Using CUDA device #: 0
[ INFO] CUDA device name: GeForce GTX TITAN X
[ INFO] Generator Loss: L1 Loss
[ INFO] ------------- iSeeBetter Network Architecture -------------
[ INFO] ----------------- Generator Architecture ------------------
...

[ INFO] Total number of parameters: 5215425
[ INFO] -----------------------------------------------------------
0%| | 0/2 [00:00<?, ?it/s]Traceback (most recent call last):
File "iSeeBetterTrain.py", line 264, in
main()
File "iSeeBetterTrain.py", line 258, in main
runningResults = trainModel(epoch, training_data_loader, netG, netD, optimizerD, optimizerG, generatorCriterion, device, args)
File "iSeeBetterTrain.py", line 61, in trainModel
next(iterTrainBar)
File "/home/amperiad/venv/lib/python3.6/site-packages/tqdm/std.py", line 1087, in iter
for obj in iterable:
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 819, in next
return self._process_data(data)
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/_utils.py", line 385, in reraise
raise self.exc_type(msg)
NotADirectoryError: Caught NotADirectoryError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/amperiad/iSeeBetter-master/dataset.py", line 204, in getitem
target, input, neigbor = load_img_future(self.image_filenames[index], self.nFrames, self.upscale_factor, self.other_dataset,self.upscale_only)
File "/home/amperiad/iSeeBetter-master/dataset.py", line 94, in load_img_future
target = modcrop(Image.open(join(filepath,'im4.png')).convert('RGB'),scale)
File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2548, in open
fp = builtins.open(filename, "rb")
NotADirectoryError: [Errno 20] Not a directory: './Vid4/foliage/001.png/im4.png'

(venv) amperiad@cuda-pc:~/iSeeBetter-master$ python3 iSeeBetterTest.py -o output.txt -c --data_dir ./Vid4 --file_list foliage_test.txt -u
Namespace(chop_forward=False, data_dir='./Vid4', debug=False, file_list='foliage_test.txt', future_frame=True, gpu_mode=True, gpus=1, model='weights/netG_epoch_4_1.pth' , model_type='RBPN', nFrames=7, other_dataset=True, output='output.txt', residual=False, seed=123, testBatchSize=1, threads=1, upscale_factor=4, upscale_only=True)
Using GPU mode
==> Loading datasets
==> Building model RBPN
[ INFO] ------------- iSeeBetter Network Architecture -------------
[ INFO] ----------------- Generator Architecture ------------------
[ INFO] DataParallel(

...
[ INFO] Total number of parameters: 12771943
Pre-trained SR model loaded from: weights/netG_epoch_4_1.pth
Traceback (most recent call last):
File "iSeeBetterTest.py", line 197, in
eval()
File "iSeeBetterTest.py", line 107, in eval
prediction = model(input, neigbor, flow)
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/amperiad/iSeeBetter-master/rbpn.py", line 82, in forward
h0 = self.DBPN(feat_input)
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/amperiad/iSeeBetter-master/dbpns.py", line 55, in forward
x = self.output(torch.cat((h3, h2, h1),1))
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/amperiad/iSeeBetter-master/base_networks.py", line 66, in forward
out = self.conv(x)
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 345, in forward
return self.conv2d_forward(input, self.weight)
File "/home/amperiad/venv/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDA out of memory. Tried to allocate 1.32 GiB (GPU 0; 11.92 GiB total capacity; 10.46 GiB already allocated; 635.38 MiB free; 280.01 MiB cached)

Using the --upscale_only switch, getting same resolution of images as input.

Thank you for your answer, but I notice you have used .mean() in your code , and I think its usage is the same as to divide by the batchsize. By the way, if it is possible to see the loss curve when you train? Thank you so much!

Hello,
MSU Graphics & Media Lab Video Group has recently launched Video Super Resolution Benchmark.
We provide a comprehensive comparison of a variety of different VSR models.
You can see our current results: MSU VSR Benchmark.
If you are interested in participating, you can add your algorithm, following the submission steps.
You can read about MSU Graphics & Media Lab Video Group here and here.

Hello, I have a problem when i running this code. I got a low PSNR's value (about 23, epoch=20, test in Vid4, 4x) and i don't know why. Could you please tell me your training set? I will appreciate it.

My training Set:

• Training dataset is Vimeo-90K
• use pretrained (RBPN_4x)
• use APITLoss
• no residual
• others use the default set

As the title of the issue says, if I run inference using the -u option I still get the same resolution output image as the input image I provide. Does -u only work for certain resolutions?

Hey, how can I use your model on my custom video?

thanks!

I specified the following settings.
--gpu mode
--gpus 2
Only GPU1 performs work, GPU2 does not occupy any memory. Thank you.

If you run python3 iSeeBetterTest.py --upscale factor, the upscale_factor==True and in the current code it never computes the psnr value and it shows me this error

Hello,

In the paper, I haven't seen any performance regarding Runtime/FPS for different sizes.

Do you have any ballpark numbers, how fast is the network for example on 360p and 720p videos?
Would it differ by a large margin if x4 upscale was changed to x2?

Best regards