sunwj / car Goto Github PK

View Code? Open in Web Editor NEW

469.0 11.0 77.0 1.2 MB

Content adaptive resampler for image downscaling

License: GNU General Public License v3.0

Python 28.09% C++ 60.64% Cuda 11.27%

car's Introduction

CAR-pytorch

Pytorch implementation of paper "Learned Image Downscaling for Upscaling using Content Adaptive Resampler"

Installation

# get CAR-pytorch source
git clone https://github.com/sunwj/CAR.git
cd CAR

# compile the code of the resampler
cd adaptive_gridsampler
python3 setup.py build_ext --inplace

Python requirements

Currently, the code only supports python3 and machine with NVIDIA GPU (and the CUDA development toolkit) installed

numpy
scipy
pytorch (== 1.3.1)
Pillow
tqdm

Pre-trained models

You can download the pre-trained models for 2x and 4x downscaling and super-resolution from here.

Inference

python3 run.py --scale 4 --img_dir path_to_images --model_dir path_to_pretrained_models \
--output_dir path_to_output

Sample results

You can download HR images of benchmark datasets, i.e., the Set5, Set14, B100 and Urban100 from here.

If you find our work useful in your research or publication, please cite our work:

Wanjie Sun, Zhenzhong Chen. "Learned Image Downscaling for Upscaling using Content Adaptive Resampler". arXiv preprint arXiv:1907.12904, 2019.

@article{sun2020learned,
  title={Learned image downscaling for upscaling using content adaptive resampler},
  author={Sun, Wanjie and Chen, Zhenzhong},
  journal={IEEE Transactions on Image Processing},
  volume={29},
  pages={4027--4040},
  year={2020},
  publisher={IEEE}
}

Acknowlegements

EDSR code is provided by thstkdgus35/EDSR-PyTorch.

car's People

Contributors

Stargazers

Watchers

Forkers

naripok 2019-paper-fun nopaixx marnie007 zeerizvee chisyliu haiboku233 dineshkumares mehmetoguzderin yukariin hdjsjyl uptodiff szwinglee moichor kitsunetic lywangpx 23vil sdukaka dynamiterdx jer7 yxyume templeblock mfromano himario johnnylu305 westamine mrpark97 howardfive arufuss tenfleques casonclagg scape1989 neuralnewtorks abitalo salonigupta11 hiddenfire21 zeta1999 yluo39github ndkamath guolong-zhang mfkiwl 0622-kitaek felipeinagaki joychopra1298 xandarin russellwmy jireh-father shnhrtkyk convexmateria hanrui56 shovelingpig timfu248 celsopitta yasho191 changetheworld20191008 ra120 nickosmooth hjgp linhuaiyi diningsystem jaedukseo cws-test sijieliu518 dabinishere akashad98 cvtuge jackzhousz jerry123jerr soonyoung-kwon barandorota shearyli gaoyabinzy leopold-fitz-ai lfliu

car's Issues

# parameters and MACs

Thank you for your research and the code.

Is it possible to share the number of parameters and MACs of this model?

Could you provide the training code?

Unable to run pre-trained model in Google Colab free GPU

I'm trying to make use of the trained model in Colab, but i get the following error during evaluation :

CUDA error at adaptive_gridsampler_kernel.cu:76 code=209(cudaErrorNoKernelImageForDevice) "cudaGetLastError()"

Colab Specs:

Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

physical_device_desc: "device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7"

Please let me know the exact specs required to run the trained model

Compilation error during adaptive_gridsampler installation

Hello,

I am trying to build and run the CAR-pytorch project on my Arch Linux machine, but I am encountering compilation errors during the installation of the adaptive_gridsampler.

Here are the steps I have followed:

Cloned the repository and navigated to the CAR directory.
Entered the adaptive_gridsampler directory and attempted to run python3 setup.py build_ext --inplace.

Initially, I encountered an error related to an incompatible version of PyTorch. The project requires PyTorch 1.3.1, but it is not available through pip or conda for Python 3.8. I tried installing PyTorch 1.4.0, but the error persisted. I then created a conda environment with Python 3.7 and installed PyTorch 1.3.1, but the compilation error still occurred.

Here is the error message I received:

/home/myuser/.conda/envs/car-env/lib/python3.7/site-packages/torch/include/pybind11/cast.h: In function ‘typename pybind11::detail::type_caster<typename pybind11::detail::intrinsic_type<T>::type>::cast_op_type<T> pybind11::detail::cast_op(make_caster<T>&)’:
/home/myuser/.conda/envs/car-env/lib/python3.7/site-packages/torch/include/pybind11/cast.h:914:120: error: expected template-name before ‘<’ token
  914 |     return caster.operator typename make_caster<T>::template cast_op_type<T>();
      |                                                                                                                        ^
/home/myuser/.conda/envs/car-env/lib/python3.7/site-packages/torch/include/pybind11/cast.h:914:120: error: expected identifier before ‘<’ token
/home/myuser/.conda/envs/car-env/lib/python3.7/site-packages/torch/include/pybind11/cast.h:914:123: error: expected primary-expression before ‘>’ token
  914 |     return caster.operator typename make_caster<T>::template cast_op_type<T>();
      |                                                                                                                           ^
/home/myuser/.conda/envs/car-env/lib/python3.7/site-packages/torch/include/pybind11/cast.h:914:126: error: expected primary-expression before ‘)’ token
  914 |     return caster.operator typename make_caster<T>::template cast_op_type<T>();
      |                                                                                                                              ^
error: command '/opt/cuda/bin/nvcc' failed with exit code 1

I would appreciate any guidance or assistance in resolving this issue. If there are any updates or additional steps required for compatibility with the latest versions of the dependencies, please let me know.

Thank you in advance for your help.

DIV2K val x4 psnr only 23.86......

Hi,
I use the model you release to test div2k val x4, but got a psnr of 23.86, which is not consistent with the benchmark in paperwithcode net.....Do I use wrong model?

Can anyone share the pretrained-model by baiduwangpan or GoogleDrive?

I can not download....Thanks soooo much.

Downscaled Images are black

I managed to get the provided models running, however when I run it the downscaled pictures are just black and therefore so are the upscaled pictures. The pictures have the correct amount of pixels and the ratio is correct. Does anyone know what might cause this issue?

Error while compiling the code of the resampler

Hey! Thank you for sharing the code.
However, when I was trying to compile the code for resampler, I faced the following error. Can you please look into it? Thanks!

(ocr) siddhant.b@gnode55:~/CAR/adaptive_gridsampler$ CUDAHOSTCXX=/usr/bin/gcc-5 python3 setup.py build_ext --inplace
running build_ext
building 'adaptive_gridsampler_cuda' extension
creating build
creating build/temp.linux-x86_64-3.7
gcc -pthread -B /home/siddhant.b/miniconda3/envs/ocr/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/siddha
nt.b/miniconda3/envs/ocr/lib/python3.7/site-packages/torch/include -I/home/siddhant.b/miniconda3/envs/ocr/lib/python3.7/site-packages/torch/include/torch/csrc/api/include
 -I/home/siddhant.b/miniconda3/envs/ocr/lib/python3.7/site-packages/torch/include/TH -I/home/siddhant.b/miniconda3/envs/ocr/lib/python3.7/site-packages/torch/include/THC 
-I/usr/local/cuda-9.0/include -I/home/siddhant.b/miniconda3/envs/ocr/include/python3.7m -c adaptive_gridsampler_cuda.cpp -o build/temp.linux-x86_64-3.7/adaptive_gridsampl
er_cuda.o -std=c++11 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=adaptive_gridsampler_cuda -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/usr/local/cuda-9.0/bin/nvcc -I/home/siddhant.b/miniconda3/envs/ocr/lib/python3.7/site-packages/torch/include -I/home/siddhant.b/miniconda3/envs/ocr/lib/python3.7/site-pa
ckages/torch/include/torch/csrc/api/include -I/home/siddhant.b/miniconda3/envs/ocr/lib/python3.7/site-packages/torch/include/TH -I/home/siddhant.b/miniconda3/envs/ocr/lib
/python3.7/site-packages/torch/include/THC -I/usr/local/cuda-9.0/include -I/home/siddhant.b/miniconda3/envs/ocr/include/python3.7m -c adaptive_gridsampler_kernel.cu -o bu
ild/temp.linux-x86_64-3.7/adaptive_gridsampler_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr
 --compiler-options '-fPIC' -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_70,code=comp
ute_70 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=adaptive_gridsampler_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
adaptive_gridsampler_kernel.cu(10): error: explicit type is missing ("int" assumed)

adaptive_gridsampler_kernel.cu(10): error: qualified name is not allowed

adaptive_gridsampler_kernel.cu(10): error: expected a ")"

adaptive_gridsampler_kernel.cu(17): warning: parsing restarts here after previous syntax error

adaptive_gridsampler_kernel.cu(72): error: class "at::Tensor" has no member "packed_accessor32"

adaptive_gridsampler_kernel.cu(72): error: type name is not allowed

adaptive_gridsampler_kernel.cu(72): error: argument list for class template "at::RestrictPtrTraits" is missing

adaptive_gridsampler_kernel.cu(72): error: expected an expression

adaptive_gridsampler_kernel.cu(72): error: class "at::Tensor" has no member "packed_accessor32"

adaptive_gridsampler_kernel.cu(72): error: type name is not allowed

adaptive_gridsampler_kernel.cu(72): error: argument list for class template "at::RestrictPtrTraits" is missing

adaptive_gridsampler_kernel.cu(72): error: expected an expression

adaptive_gridsampler_kernel.cu(73): error: class "at::Tensor" has no member "packed_accessor32"

adaptive_gridsampler_kernel.cu(73): error: type name is not allowed

adaptive_gridsampler_kernel.cu(73): error: argument list for class template "at::RestrictPtrTraits" is missing

adaptive_gridsampler_kernel.cu(73): error: expected an expression

adaptive_gridsampler_kernel.cu(72): error: class "at::Tensor" has no member "packed_accessor32"

adaptive_gridsampler_kernel.cu(72): error: type name is not allowed

adaptive_gridsampler_kernel.cu(72): error: argument list for class template "at::RestrictPtrTraits" is missing

adaptive_gridsampler_kernel.cu(72): error: expected an expression

adaptive_gridsampler_kernel.cu(73): error: class "at::Tensor" has no member "packed_accessor32"

adaptive_gridsampler_kernel.cu(73): error: type name is not allowed

adaptive_gridsampler_kernel.cu(73): error: argument list for class template "at::RestrictPtrTraits" is missing

adaptive_gridsampler_kernel.cu(73): error: expected an expression

adaptive_gridsampler_kernel.cu(73): error: class "at::Tensor" has no member "packed_accessor32"

adaptive_gridsampler_kernel.cu(73): error: type name is not allowed

adaptive_gridsampler_kernel.cu(73): error: argument list for class template "at::RestrictPtrTraits" is missing

adaptive_gridsampler_kernel.cu(73): error: expected an expression

adaptive_gridsampler_kernel.cu(74): error: class "at::Tensor" has no member "packed_accessor32"

adaptive_gridsampler_kernel.cu(74): error: type name is not allowed

adaptive_gridsampler_kernel.cu(74): error: argument list for class template "at::RestrictPtrTraits" is missing

adaptive_gridsampler_kernel.cu(74): error: expected an expression

adaptive_gridsampler_kernel.cu(71): error: no instance of function template "kernel_adaptive_gridsampler_update_output" matches the argument list
            argument types are: (<error-type>, int, <error-type>, <error-type>, int, <error-type>, <error-type>, int, <error-type>, <error-type>, int, <error-type>, const int, const int, <error-type>, int, <error-type>, int64_t)

24 errors detected in the compilation of "/tmp/tmpxft_00004ba5_00000000-8_adaptive_gridsampler_kernel.compute_70.cpp1.ii".
error: command '/usr/local/cuda-9.0/bin/nvcc' failed with exit status 1
(ocr) siddhant.b@gnode55:~/CAR/adaptive_gridsampler$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
(ocr) siddhant.b@gnode55:~/CAR/adaptive_gridsampler$ gcc --version
gcc (Ubuntu 5.5.0-12ubuntu1~16.04) 5.5.0 20171010
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(ocr) siddhant.b@gnode55:~/CAR/adaptive_gridsampler$ g++ --version
g++ (Ubuntu 5.5.0-12ubuntu1~16.04) 5.5.0 20171010
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I even tried suggestions from here, but it did not help.

ModuleNotFoundError: No module named 'adaptive_gridsampler.adaptive_gridsampler_cuda'

For the inference operation, the following operation was carried out, but an error appears.

get CAR-pytorch source

git clone https://github.com/sunwj/CAR.git
cd CAR

compile the code of the resampler

cd adaptive_gridsampler
python3 setup.py build_ext --inplace

required version to compile cuda code

can someone post a workable version of pytorch, cuda, c++ that can compile the code?

Can I get your train code or backward code of GridSampler?

Can I get your train code or backward code of GridSampler? Thx

Can I

Possible to infer without HR?

Thank you for posting your code. I noticed the run.py command only works if you already have the original, high resolution image. Is it possible to infer with a smaller source image and produce a high res output?

Training code request

Can you please let me know how to get/run the training code?

Training Codes

Dear author:
I want to train this model on different datasets and compare the results. Unfortunately, there is no code for training. Can you please add training codes, too?
thanks a lot

Error when running the code

Hi @sunwj, thanks for open sourcing the code!

I tried to run it and have successfully setup the dependencies. However, when running run.py, it gives this error message
CUDA error at adaptive_gridsampler_kernel.cu:76 code=48(cudaErrorNoKernelImageForDevice) "cudaGetLastError()"

And the output is a black image and with (Mean PSNR: 6.53; Mean SSIM: 0.1514). Wondering if there's a CPU version just for model inference? That would be less likely to encounter this kind of environmental error!

CAN I GET TRAIN CODE?

Plz

CUDA error at adaptive_gridsampler_kernel.cu:76 code=209(cudaErrorNoKernelImageForDevice) "cudaGetLastError()"

I get the subject error and then the out image is just black. What can be the issue here?

Adaptive gridsample compilation error

Hi,

I am having trouble compiling the adaptive_gridsampler. The error message is very long and goes past the limit saved by my terminal window, but seems to repeat itself. Here is the error:

 class ExpandingArray {
       ^~~~~~~~~~~~~~
In file included from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/c10/core/ScalarType.h:6:0,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/c10/core/Scalar.h:9,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/ATen/core/TensorBody.h:7,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/ATen/Tensor.h:11,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/ATen/Context.h:4,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/ATen/ATen.h:5,
                 from adaptive_gridsampler_cuda.cpp:1:
/home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/c10/util/Optional.h: In instantiation of ‘c10::optional<T>& c10::optional<T>::operator=(c10::optional<T>&&) [with T = torch::ExpandingArray<2>]’:
/home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/options/pooling.h:428:8:   required from ‘void torch::nn::Cloneable<Derived>::clone_(torch::nn::Module&, const c10::optional<c10::Device>&) [with Derived = torch::nn::FractionalMaxPool2dImpl]’
/home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/optim/sgd.h:48:17:   required from here
/home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/c10/util/Optional.h:396:23: error: passing ‘const torch::ExpandingArray<2>’ as ‘this’ argument discards qualifiers [-fpermissive]
       contained_val() = std::move(*rhs);
       ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
In file included from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/options/conv.h:6:0,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/functional/conv.h:3,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/functional.h:4,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:4,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/extension.h:4,
                 from adaptive_gridsampler_cuda.cpp:2:
/home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/expanding_array.h:23:7: note:   in call to ‘torch::ExpandingArray<2>& torch::ExpandingArray<2>::operator=(const torch::ExpandingArray<2>&)’
 class ExpandingArray {
       ^~~~~~~~~~~~~~
In file included from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/c10/core/ScalarType.h:6:0,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/c10/core/Scalar.h:9,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/ATen/core/TensorBody.h:7,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/ATen/Tensor.h:11,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/ATen/Context.h:4,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/ATen/ATen.h:5,
                 from adaptive_gridsampler_cuda.cpp:1:
/home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/c10/util/Optional.h: In instantiation of ‘c10::optional<T>& c10::optional<T>::operator=(c10::optional<T>&&) [with T = torch::ExpandingArray<2, double>]’:
/home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/options/pooling.h:428:8:   required from ‘void torch::nn::Cloneable<Derived>::clone_(torch::nn::Module&, const c10::optional<c10::Device>&) [with Derived = torch::nn::FractionalMaxPool2dImpl]’
/home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/optim/sgd.h:48:17:   required from here
/home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/c10/util/Optional.h:396:23: error: passing ‘const torch::ExpandingArray<2, double>’ as ‘this’ argument discards qualifiers [-fpermissive]
       contained_val() = std::move(*rhs);
       ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
In file included from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/options/conv.h:6:0,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/functional/conv.h:3,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/functional.h:4,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:4,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                 from /home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/extension.h:4,
                 from adaptive_gridsampler_cuda.cpp:2:
/home/jamespb/Projects/CAR/car_venv/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/expanding_array.h:23:7: note:   in call to ‘torch::ExpandingArray<2, double>& torch::ExpandingArray<2, double>::operator=(const torch::ExpandingArray<2, double>&)’
 class ExpandingArray {
       ^~~~~~~~~~~~~~
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Additional info:
OS: Ubuntu 18.04
gcc: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
python packages:
Package Version
--------------- -----------------
cycler 0.10.0
decorator 4.4.2
future 0.18.2
imagecodecs 2020.2.18
imageio 2.8.0
kiwisolver 1.2.0
matplotlib 3.2.1
networkx 2.4
numpy 1.18.4
Pillow 7.1.2
pip 20.1.1
pkg-resources 0.0.0
pyparsing 2.4.7
python-dateutil 2.8.1
PyWavelets 1.1.1
scikit-image 0.17.2
scipy 1.4.1
setuptools 39.0.1
six 1.14.0
tifffile 2020.5.11
torch 1.5.0
torch-nightly 1.2.0.dev20190805
tqdm 4.46.0
wheel 0.34.2

Thanks, any help is much appreciated

VRAM Limitation for High Resolution images

Hi @sunwj . Thank you for providing this repository. I was very happy to try it on high-resolution images ( resolution > = 16 Megapixels) and noticed the GPU was running out of memory. I went back and tried it with the benchmark datasets and noticed that images with 0.6 megapixels (1024 x 640 pixels) would consume 5 GB VRAM. What could we do to reduce the VRAM requirements? Is this relationship linear with size of the image?

When I set learning rate to 10e-4, graident values are diverged. So I use gradient clipping and set the value to 0.01.
I wonder if it is reasonable value.
What value did you set to tuning parameter alpha?

thank you in advance!