hyeongminlee / adacof-pytorch Goto Github PK

Official source code for our paper "AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation" (CVPR 2020)

License: MIT License

Python 100.00%

video-frame-interpolation deep-learning pytorch cuda cupy python

adacof-pytorch's Introduction

Hyeongmin Lee

I am an ML Research Scientist at Twelve Labs. I’m interested in exploring the fundamental nature of Images & Videos and any technology for Representing & Enhancing them.
To know more about me, please visit my Personal Website
Thank you :)

Contact

E-mail: [email protected]
Web: https://hyeongminlee.github.io
Github: https://github.com/HyeongminLEE
Youtube: https://www.youtube.com/channel/UCz14hRj69FfGsl7ftF0D3ig

Research Interest

Computer Vision & Deep Learning / Video Processing / Video Enhancement / Video Understanding / Image & Video Compression

Skill

Python, CUDA, Visual C/C++, Matlab
Pytorch, Tensorflow

adacof-pytorch's People

Contributors

Stargazers

Watchers

adacof-pytorch's Issues

AdaCoF Operation

Hi, when you were writing the code for AdaCoF in cupy, what software or resources did you use for that, I am trying to understand the cupy/C code, but am currently not sure what to use for debugging to check the outputs at each stage
Thanks

only interpolate one frame?

in interpolate_video.py , is the arg of '--dilation' useless? I want to inpterpolate 4 frame , such as from 30fps to 120fps. Could you tell me what should I do. thanks

Insert multiple frames

Thank you very much for sharing your project. Can I insert any number of Frame between two Frame ?(such as 2,3,4,5...)

Ask about Oclussion Map Visualization

As seen in Figure 6 in your paper, may i ask how was your occlusion map visualized?

Also, AdaCoF is a great paper, thank you for your hard work!

Ask the meaning of variable Alpha1 in the code (horizontal or vertical?)

In your implementation, does variable Alpha1 refer to horizontal sampling or vertical sampling of an image?

Loss function question

Hi, the default loss function is set to 1Charb+0.01g_Spatial+0.005*g_Occlusion'. Charb is the Charbonnier loss, but I can't seem to figure out what g_spatial and g_occlusion are based on the code?
For the adversarial loss, of the options stated (GAN, WGAN, WGAN_GP, T_WGAN_GP, FI_GAN) which was the one used to get the best results in the paper?
Thanks for your help

How do I export onnx ? onnx does not support adACof operations

About implemention of "backward operation" of the input

Dear author,
Thanks for your great work, it is very inspiring! I have a question about the implementation of the ''backward operation'' of the input. Would you mind implementing the ''backward operation'' and releasing the codes? It would be very helpful to us.
Looking forward to your reply!

Best regards!
yung

evaluation index

Can you share the code files about the evaluation indicators SSIM, IE, NIE？Thank you！

About gan loss finetune.

hello, i see the pretrained model in this repository has not trained with vgg loss and adv loss. in your paper, you finetune the model by λ1 = 0.01; λ = 1; λadv = 0.005, so i have tried to finetune the base model with --loss '0.01Charb+0.01g_Spatial+0.005g_Occlusion+1.0VGG+0.005*GAN', but when it undergo the 10st epoch, the model can not run a stable and good performance, what parameters did you set for the finetune training ?

Can you share code about occlusion, meanflow and varflow maps.I'm really a little stupid

FileNotFoundError: [Errno 2] No such file or directory: '/mnt/vimeo_triplet/Beanbags/frame10.png'

(AdaCof) root@e82d0083b966:/mnt/AdaCoF# python train.py --train /mnt/vimeo_triplet --out_dir /mnt/AdaCoF/output
1.000 * Charb
0.010 * g_Spatial
0.005 * g_Occlusion
Traceback (most recent call last):
File "train.py", line 82, in
main()
File "train.py", line 65, in main
my_trainer = Trainer(args, train_loader, TestDB, model, loss, start_epoch)
NameError: name 'TestDB' is not defined
(AdaCof) root@e82d0083b966:/mnt/AdaCoF# python train.py --train /mnt/vimeo_triplet --out_dir /mnt/AdaCoF/output
Traceback (most recent call last):
File "train.py", line 82, in
main()
File "train.py", line 54, in main
TestDB = Middlebury_other(args.test_input, args.gt)
File "/mnt/AdaCoF/TestModule.py", line 39, in init
self.input0_list.append(to_variable(self.transform(Image.open(input_dir + '/' + item + '/frame10.png')).unsqueeze(0)))
File "/opt/conda/envs/AdaCof/lib/python3.7/site-packages/PIL/Image.py", line 2975, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/vimeo_triplet/Beanbags/frame10.png'

I get very weird flickering in the output of the arbitrary video.

Here is the link to the video: https://drive.google.com/file/d/1fshHn87HQRNUp0YH6LTCl5du5n3ChA6q/view?usp=sharing
Is it due to dataset bias and will not generalise in the real use cases?

about dataset

Hello,I see sentences about datasets in your paper.
These sentences are'Training Dataset. We use Vimeo90K dataset for training. It contains 51,312 triplets of 256 × 448 video frames.'
But when i training with the ‘vimeo_triplet’ folder ,i find it contains 73191 triplets.
I want to ask you, if the number of training datasets is different from you, what are the adverse effects？
thank you!

About occlusion map

Hi~! In your paper, the element of occlusion map is either 0 (implies the pixel is visible only in I_(n+1)) or 1(implies the pixel is visible only in I_n). How do you know which pixels are visible in both input frames, as your paper section "offset visualization" shows?

About the 1280 × 720 resolution！

Dear author, I want to use your model to deal with 1624 × 1224 resolution video, but unfortunately, your model seems to limit the maximum resolution to 1280 × 720。 How can you change your code to deal with higher resolution video? I'm a little worried. If you see the message, please contact my email: [email protected]

Error when doing backward computation using adacof.py.

Thank you very much for your adacof.py algorithm. When I use it in my work, the forward computing is fine, however, when calling loss.backward(), I met the following errors:

  File "/home/gph/Desktop/rpg_e2vid_internal-master/e2vid_pytorch/trainer/lstm_trainer.py", line 265, in _train_epoch
    loss.backward()
  File "/home/gph/anaconda3/envs/py38_torch171/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/gph/anaconda3/envs/py38_torch171/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([2, 512, 14, 14], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(512, 512, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams 
    data_type = CUDNN_DATA_FLOAT
    padding = [1, 1, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x7f06540b6820
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 2, 512, 14, 14, 
    strideA = 100352, 196, 14, 1, 
output: TensorDescriptor 0x7f06540bd860
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 2, 512, 14, 14, 
    strideA = 100352, 196, 14, 1, 
weight: FilterDescriptor 0x7f06540ad110
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 512, 512, 3, 3, 
Pointer addresses: 
    input: 0x7f06e9519000
    output: 0x7f06e9800000
    weight: 0x7f09eaa20000
Additional pointer addresses: 
    grad_output: 0x7f06e9800000
    grad_input: 0x7f06e9519000
Backward data algorithm: 4

Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

Process finished with exit code 1

Any suggestions? Thanks!