Welcome to the Pytorch implementation of paper "SinMPI: Novel View Synthesis from a Single Image with Expanded Multiplane Images" (SIGGRAPH Asia 2023).
(1) Create a new conda environment specified in requirements.txt.
(2) Download pretrained weights of depth-aware inpainter ecweights and put them in to 'warpback/ecweights/xxx.pth'.
sh scripts/train_all.sh
This demo converts 'test_images/Syndney.jpg' to an expanded MPI and renders novel views as in 'ckpts/Exp-Syndney-new/MPI_rendered_views.mp4'.
In the above demo, we specify 'test_images/Syndney.jpg'
as the input image, then we continuously outpaint the input image:
CUDA_VISIBLE_DEVICES=$cuda python outpaint_rgbd.py \
--width $width \
--height $height \
--ckpt_path $ckpt_path \
--img_path $img_path \
--extrapolate_times $extrapolate_times
Then we get the outpainted image and its depth estimated by a monocular depth estimator (DPT):
CUDA_VISIBLE_DEVICES=$cuda python train_inpainting.py \
--width $width \
--height $height \
--ckpt_path $ckpt_path \
--img_path $img_path \
--num_epochs 10 \
--extrapolate_times $extrapolate_times \
--batch_size 1 #--load_warp_pairs --debugging
CUDA_VISIBLE_DEVICES=$cuda python train_mpi.py \
--width $width \
--height $height \
--ckpt_path $ckpt_path \
--img_path $img_path \
--num_epochs 10 \
--extrapolate_times $extrapolate_times \
--batch_size 1 #--debugging #--resume
After optimization, we render novel views:
Notice the above demo is designed for fast illustration (FPS is low). For better quality:
Increasing the sample rate and sample areas helps to optimize MPI with better quality. To add training and rendering view trajectories, modify 'dataloaders/single_img_data.py'.
If you find our work helpful, please cite our paper. Thank you!
ACM Reference Format:
Guo Pu, Peng-Shuai Wang, and Zhouhui Lian. 2023. SinMPI: Novel View
Synthesis from a Single Image with Expanded Multiplane Images. In SIGGRAPH Asia 2023 Conference Papers (SA Conference Papers '23), December
12–15, 2023, Sydney, NSW, Australia. ACM, New York, NY, USA, 10 pages.
https://doi.org/10.1145/3610548.3618155
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent