Scene Dynamic Recovery (SDR) by inverting distorted Rolling shutter (RS) images to an undistorted high frame-rate Global Shutter (GS) video is a severely ill-posed problem due to the missing temporal dynamic information in both RS intra-frame scanlines and inter-frame exposures, particularly when prior knowledge about camera/object motions is unavailable. Commonly used artificial assumptions on scenes/motions and data-specific characteristics are prone to producing sub-optimal solutions in real-world scenarios. To address this challenge, we propose an event-based SDR network within a self-supervised learning paradigm, i.e., SelfUnroll. We leverage the extremely high temporal resolution of event cameras to provide accurate inter/intra-frame dynamic information. Specifically, an Event-based Inter/intra-frame Compensator (E-IC) is proposed to predict the per-pixel dynamic between arbitrary time intervals, including the temporal transition and spatial translation. Exploring connections in terms of RS-RS, RS-GS, and GS-RS, we explicitly formulate mutual constraints with the proposed E-IC, resulting in supervisions without ground-truth GS images. Extensive evaluations over synthetic and real datasets demonstrate that the proposed method achieves state-of-the-art and shows remarkable performance for event-based RS2GS inversion in real-world scenarios.

• Python 3.8.0
• Pytorch 1.11.0
• NVIDIA GPU + CUDA 11.2
• numpy, argparse

You can create a new Anaconda environment as follows.

conda create -n selfunroll python=3.8
conda activate selfunroll

Clone this repository.

git clone [email protected]:w3un/selfunroll_code.git

Install the above dependencies.

cd selfunroll
pip install -r requirements.txt

Pretrained models can be downloaded via Google Drive.
In our paper, we conduct experiments on three types of data:

• Fastec-RS contains synthetic RS images from DeeplUnrollNet. We first convert low frame rate into high frame rate GS videos using Superslomo, and then generate events by V2E.
• Gev-RS contains synthetic RS images and synthetic events from EvUnroll, where RS images are generated using the same manner as Fastec-RS.
• Gev-Real-RS contains real-world RS images and real-world events from EvUnroll.
• We built a real-world dataset DRE which contains real-world RS images and real-world events. (The data is coming soon.)

• Change the parent directory

cd selfunroll_code

• Copy the pretrained model to directory './PreTrained/'

cd script

(Please replace the parameter "RSpath" with the place where the dataset is downloaded.)

• Test on Fastec data

python test.py --Dataset Fastec --RSpath path_of_data  --test_unroll_path ../PreTrained/fast.pth --result_path ../result/fastec/ 

• Test on Gev data

python test.py --Dataset Gev --RSpath path_of_data --test_unroll_path ../PreTrained/gev.pth --result_path ../result/gev/  

• Test on Gev-Real data

python test.py --Dataset Gevreal --RSpath path_of_data --test_unroll_path ../PreTrained/gevreal.pth --result_path ../result/gevreal/ 

• Test on DRE data

python test.py --Dataset Drereal --RSpath path_of_data --test_unroll_path ../PreTrained/dre.pth --result_path ../result/dre/ 

Main Parameters:

• --Dataset : dataset name.
• --RSpath : path of dataset.
• --test_unroll_path : path of pretrain model.
• --save_path : path of reconstruction results.
• --target : timestamp of target GS frame (0/1 represents the exposure time of first/last scanline).

If you find our work useful in your research, please cite:

@article{wang2023self,
        title={Self-Supervised Scene Dynamic Recovery from Rolling Shutter Images and Events},
        author={Wang, Yangguang and Zhang, Xiang and Lin, Mingyuan and Yu, Lei and Shi, Boxin and Yang, Wen and Xia, Gui-Song},
        journal={arXiv},
        year={2023}
        }