Code Monkey home page Code Monkey logo

anim-nerf's Introduction

Animatable Neural Radiance Fields from Monocular RGB Videos

overview

Animatable Neural Radiance Fields from Monocular RGB Videos
Jianchuan Chen, Ying Zhang, Di Kang, Xuefei Zhe, Linchao Bao, Xu Jia, Huchuan Lu

Demos

  • male-3-casual

More demos please see Demos.

Requirements

For visualization

  • pyrender
  • Trimesh
  • PyMCubes

Run the following code to install all pip packages:

pip install -r requirements.txt

To install KNN_CUDA, we provide two ways:

  • from source
    git clone https://github.com/unlimblue/KNN_CUDA.git
    cd KNN_CUDA
    make && make install
  • from wheel
    pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl

SMPL models

To download the SMPL model go to this (male, female and neutral models).

Place them as following:

smplx
└── models
    └── smpl
        ├── SMPL_FEMALE.pkl
        ├── SMPL_MALE.pkl
        └── SMPL_NEUTRAL.pkl

Data Preparation

Thanks to @radman for providing the Colab for data preparation at Here.

People-Snapshot datasets

  • Prepare images and smpls
    python -m tools.people_snapshot --data_root ${path_to_people_snapshot_datasets} --people_ID male-3-casual --gender male --output_dir data/people_snapshot
  • Prepare template
    python -m tools.prepare_template --data_root data/people_snapshot --people_ID male-3-casual --model_type smpl --gender male --model_path ${path_to_smpl_models}

iPER datasets or Custom datasets

  • Prepare images

    First convert video to images, and crop the images to make the person as centered as possible.

    python -m tools.video_to_images --vid_file ${path_to_iper_datasets}/023_1_1.mp4 --output_folder data/iper/iper_023_1_1/cam000/images --img_wh 800 800 --offsets -30 20
  • Images segmentation

    Here, we use RVM to extact the foreground mask of the person.

    python -m tools.rvm --images_folder data/iper/iper_023_1_1/cam000/images --output_folder data/iper/iper_023_1_1/cam000/images
  • SMPL estimation

    In our experiment, we use VIBE to estimate the smpl parameters.

    python -m tools.vibe --images_folder data/iper/iper_023_1_1/cam000/images --output_folder data/iper/iper_023_1_1

    Then convert vibe_output.pkl to the format in our experiment setup.

    python -m tools.convert_vibe --data_root data/iper --people_ID iper_023_1_1 --gender neutral
  • Prepare template

    python -m tools.prepare_template --data_root data/iper --people_ID iper_023_1_1 --model_type smpl --gender neutral --model_path ${path_to_smpl_models}

Training

  • Training on the training frames
    python train.py --cfg_file configs/people_snapshot/male-3-casual.yml
  • Finetuning the smpl params on the testing frames
    python train.py --cfg_file configs/people_snapshot/male-3-casual_refine.yml train.ckpt_path checkpoints/male-3-casual/last.ckpt

We provide the preprocessed data and pretrained models at Here

Visualization

Novel view synthesis

python novel_view.py --ckpt_path checkpoints/male-3-casual/last.ckpt

3D reconstruction

python extract_mesh.py --ckpt_path checkpoints/male-3-casual/last.ckpt

Shape Editing

  • To make the person look fatter, we can properly reduce the betas_2th parameter.
    python novel_view.py --ckpt_path checkpoints/male-3-casual/last.ckpt --betas_2th -2.0
  • In contrast, we can increase the betas_2th parameter to make the person look slimmer.
    python novel_view.py --ckpt_path checkpoints/male-3-casual/last.ckpt --betas_2th 3.0

Novel pose synthesis

python novel_pose.py --ckpt_path checkpoints/male-3-casual/last.ckpt

The mixamo motion capture smpl parameters can be downloaded from here.

Testing

python test.py --ckpt_path checkpoints/male-3-casual_refine/last.ckpt --vis

Citation

If you find the code useful, please cite:

@misc{chen2021animatable,
      title={Animatable Neural Radiance Fields from Monocular RGB Videos}, 
      author={Jianchuan Chen and Ying Zhang and Di Kang and Xuefei Zhe and Linchao Bao and Xu Jia and Huchuan Lu},
      year={2021},
      eprint={2106.13629},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgements

Parts of the code were based on from kwea123's NeRF implementation: https://github.com/kwea123/nerf_pl. Some functions are borrowed from PixelNeRF https://github.com/sxyu/pixel-nerf

anim-nerf's People

Contributors

janaldochen avatar woo1 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.