Code and data for CVPR 2018 paper: Single Image Reflection Removal with Perceptual Losses

Project Website | Paper

This code is based on tensorflow. It has been tested on Ubuntu 16.04 LTS.

• Clone/Download this repo
• $ cd perceptual-reflection-removal
• $ mkdir VGG_Model
• Download VGG-19. Search imagenet-vgg-verydeep-19 in this page and download imagenet-vgg-verydeep-19.mat. We need the pre-trained VGG-19 model for our hypercolumn input and feature loss
• move the downloaded vgg model to folder VGG_Model

A minimal conda environment to test the pretrained model is provided. (credit to @ironbar)

conda env create -f env.yml

Both synthetic and real dataset are now available here.

• synthetically blended data is created on-the-fly so there is only reflection layer and transmission layer folders
• real data has blended images and ground truth transmission layer (NO ground truth reflection layer, reason is detailed in the paper)
• real images used for test have indices: [3, 4, 9, 12, 15, 22, 23, 25, 29, 39, 46, 47, 58, 86, 87, 89, 93, 103, 107, 110]
  • special thanks to @Vandermode who helped with sorting out dataset-related problems

You can also try with your own dataset. For example, to generate your own synthetic dataset, prepare for two folders of images, one used for transmission layer and the other used for reflection layer. In order to use our data loader, please follow the instructions below to organize your dataset directories. Overall, we name the directory of input image with reflection blended, the ground truth transmission layer transmission_layer and the ground truth reflection layer reflection_layer.

For synthetic data, since we generate images on the fly, there is no need to have the blended subdirectory.

+-- root_training_synthetic_data

+-- reflection_layer
+-- transmission_layer

For real data, since the ground truth reflection layer is not available, there is no need to have the reflection_layer subdirectory.

+-- root_training_real_data

+-- blended
+-- transmission_layer

(please have yous real and synthetic data paths set up correctly as described above):

python3 main.py --data_syn_dir your_syn_data_path --data_real_dir your_real_data_path for triaining from scratch

python3 main.py --continue_training --data_syn_dir your_syn_data_path --data_real_dir your_real_data_path --task your_checkpoint_path for training with existing checkpoint (checkpoint path specified by the --task argument)

--task: the checkpoint directory path. For example, for --task experiment_1, the checkpoints (e.g. *.model) are saved inside ./experiment_1/

--data_syn_dir: root path to the images to generate synthetic data

--data_real_dir: root path to the real images

--save_model_freq: frequency to save model and the output images

--is_hyper: whether to use hypercolumn features as input, all our trained models uses hypercolumn features as input

• Download pre-trained model here
• $ tar -xvzf pre-trained.tar.gz
• this should extract the models into a newly created folder called pre-trained
• Change test_path (line 419) to your test image folder. If you want to test on the provided test images (e.g. in ./test_images/real/), keep it as it is.
• test results can be found in ./test_results/

Then, run

$ python3 main.py --task pre-trained --is_training 0

Part of the code is based upon FastImageProcessing

If you find this work useful for your research, please cite:

@inproceedings{zhang2018single,
  title = {Single Image Reflection Separation with Perceptual Losses},
  author = {Zhang, Xuaner and Ng, Ren and Chen, Qifeng}
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
  year = {2018}
}

Please contact me if there is any question (Cecilia Zhang [email protected])