This repository contains the source code of video synthesis project developed by Xipeng Xie, Nikita Lockshin, and LianFeng Li. This project is inspired by Monkey-Net from Siarohin et al. and by Mask-RCNN from Abdulla et al. We propose a method to animate multiple objects from the source image to follow another object's motion pattern in a driving video by utilizing deep motion transfer.

Source	SourceRPN	Driving	Generated

Source	SourceRPN	Driving	Generated

python run_all_mgif.py --config config/moving-gif.yaml --driving_video sup-mat/driving.png --checkpoint path/to/checkpoint --image sup-mat/target2.png --image_shape 256,128

Source	SourceRPN	Driving	Generated

python run_all_taichi.py --config config/taichi.yaml --driving_video Images/TaiChi_Driving.gif --checkpoint path/to/checkpoint --image Images/P2TaiChi_Source.png --image_shape 256,256

Source	Driving	Generated

Download the checkpoint first from here

cd new-monkey-net
python demo.py --config  config/taichi.yaml --driving_video ../sup-mat/00001050.png --source_image sup-mat/64.jpg --checkpoint <path/to/checkpoint> --image_shape 64,64

cd new-monkey-net
CUDA_VISIBLE_DEVICES=0 python run.py --config config/dataset_name.yaml

pip install -r requirements.txt
cd Mask_RCNN
pip3 install -r requirements.txt
python3 setup.py install

python find_rois.py --image <path to input image>

To run a demo, download a checkpoint (more checkpoint we get can be checked HERE) and run the following command:

python demo.py --config config/moving-gif.yaml --checkpoint <path/to/checkpoint>

The result will be stored in demo.gif.

python demo.py --i_am_iddo_drori True --config config/moving-gif.yaml --checkpoint <path/to/checkpoint>

To train a model on specific dataset run:

CUDA_VISIBLE_DEVICES=0 python run.py --config config/dataset_name.yaml

The command will create a folder in the log directory (each run will create a time-stamped new directory). Checkpoints will be saved to this folder. To check the loss values during training in see log.txt. You can also check training data reconstructions in the train-vis subfolder.

1. Shapes. This dataset is saved along with repository. Download the checkpoint. Training takes about 17 minutes in Colab.

2. Actions. This dataset is also saved along with repository. And training takes about 1 hour.

3. Tai-chi. Becauce of copyright, the dataset can't be public, please contact the author if you need it. Download the checkpoint. Training takes about 34 hours, on 1 gpu.

4. MGif. The preprocessed version of this dataset can be downloaded. Check for details on this dataset. Download the checkpoint. Training takes about 10 hours, on 1 gpu.