ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

Duolikun Danier, Fan Zhang, David Bull

Accepted in CVPR 2022.

Dependencies and Installation

The following packages were used to evaluate the model.

python==3.8.8
pytorch==1.7.1
torchvision==0.8.2
cudatoolkit==10.1.243
opencv-python==4.5.1.48
numpy==1.19.2
pillow==8.1.2
cupy==9.0.0
sk-video==1.1.10

Installation with anaconda:

conda create -n stmfnet python=3.8.8
conda activate stmfnet
conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 -c pytorch
conda install -c conda-forge cupy
pip install opencv-python==4.5.1.48
pip install sk-video==1.1.10

Model

Preparing datasets

Training sets:

[Vimeo-90K] | [BVI-DVC quintuplets]

Test sets:

[UCF101] | [DAVIS] | [SNU-FILM] | [VFITex]

The dataset folder names should be lower-case and structured as follows.

└──── <data directory>/
    ├──── ucf101/
    |   ├──── 0/
    |   ├──── 1/
    |   ├──── ...
    |   └──── 99/
    ├──── davis90/
    |   ├──── bear/
    |   ├──── bike-packing/
    |   ├──── ...
    |   └──── walking/
    ├──── snufilm/
    |   ├──── test-easy.txt
    |   ├──── test-medium.txt
    |   ├──── test-hard.txt
    |   ├──── test-extreme.txt
    |   └──── data/SNU-FILM/test/...
    ├──── vfitex/
    |   ├──── beach02_4K_mitch/
    |   ├──── bluewater_4K_pexels/
    |   ├──── ...
    |   └──── waterfall_4K_pexels/
    ├──── bvidvc/quintuplets
    |   ├──── 00000/
    |   ├──── 00001/
    |   ├──── ...
    |   └──── 17599/
    └──── vimeo_septuplet/
        ├──── sequences/
        ├──── readme/
        ├──── sep_testlist.txt
        └──── sep_trainlist.txt

Downloading the pre-trained model

Download the pre-trained ST-MFNet from here.

Evaluation (on test sets)

python evaluate.py \
--net STMFNet \
--data_dir <data directory> \
--checkpoint <path to pre-trained model (.pth file)> \
--out_dir eval_results \
--dataset <dataset name>

where <dataset name> should be the same as the class names defined in data/testsets.py, e.g. Snufilm_extreme_quintuplet.

Evaluation (on videos)

python interpolate_yuv.py \
--net STMFNet \
--checkpoint <path to pre-trained model (.pth file)> \
--yuv_path <path to input YUV file> \
--size <spatial size of input YUV file, e.g. 1920x1080>
--out_fps <output FPS, e.g. 60>
--out_dir <desired output dir>

See more details in interpolate_yuv.py. Note the script provided is for up-sampling .yuv files. To process .mp4 files, one can modify the frame reading parts of the script, or simply convert mp4 to yuv using ffmpeg then use this script.

Training

Feel free to experiment with other options, but here is an example:

python train.py \
--net STMFNet \
--data_dir <data directory> \
--out_dir ./train_results \
--epochs 70 \
--batch_size 4 \
--loss 1*Lap \
--patch_size 256 \
--lr 0.001 \
--decay_type plateau \
--gamma 0.5 \
--patience 5 \
--optimizer ADAMax

Example results

Citation

@misc{danier2021spatiotemporal,
     title={ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation}, 
     author={Duolikun Danier and Fan Zhang and David Bull},
     year={2021},
     eprint={2111.15483},
     archivePrefix={arXiv},
     primaryClass={cs.CV}
}

Acknowledgement

Lots of code in this repository are adapted/taken from the following repositories:

We would like to thank the authors for sharing their code.

cv-ip / st-mfnet Goto Github PK

st-mfnet's Introduction