[Project website] [Dataset] [Video]
Clothing plays a fundamental role in digital humans. Current approaches to animate 3D garments are mostly based on realistic physics simulation, however, they typically suffer from two main issues: high computational run-time cost, which hinders their development; and simulation-to-real gap, which impedes the synthesis of specific real-world cloth samples. To circumvent both issues we propose PERGAMO, a data-driven approach to learn a deformable model for 3D garments from monocular images. To this end, we first introduce a novel method to reconstruct the 3D geometry of garments from a single image, and use it to build a dataset of clothing from monocular videos. We use these 3D reconstructions to train a regression model that accurately predicts how the garment deforms as a function of the underlying body pose. We show that our method is capable of producing garment animations that match the real-world behaviour, and generalizes to unseen body motions extracted from motion capture dataset.
IGL only supports and recommends the use of Anaconda. However, the environment can be set up using only pip
by installing the IGL bindings from source.
The general steps are as follows:
- Install PyTorch according to your system ( https://pytorch.org/get-started/locally/ )
- See the
requirements.txtfile to check the needed packages- This is usually done with
pip install -r requirements.txt, but Anaconda may have a different way of doing things
- This is usually done with
- Install IGL bindings ( https://github.com/libigl/libigl-python-bindings )
- Install Kaolin ( https://kaolin.readthedocs.io/en/latest/notes/installation.html )
- You can download the weights from
OneDrive
. Place the
weightsfolder from OneDrive into thedatafolder of this repository. - PERGAMO needs SMPL. You can download it from SMPL. Rename the file from
basicmodel_neutral_lbs_10_207_0_v1.1.0.pkltosmpl_neutral.pkland save it underdata/smpl/.
To run the reconstruction, please check out run_recons.sh.
To run the regression, there are 2 sets of 3 scripts. Please check out run_regression.sh to see how it works.
The output is generated under data (test_sequence for AMASS scripts, train/validation_sequence for reconstructed
scripts).
To visualize using Blender, load the .obj file with the option Geometry > Keep Vert Order. Then, add a Mesh Cache
modifier to the loaded mesh. Change the type to PC2 and then load the .pc2 file adjacent to the .obj.
You can download a dataset from OneDrive .
Each data set has the following folder hierarchy:
DataDanXXXXX
├─ clips (video files)
| ├─ dan-X01.mp4
| ├─ dan-X02.mp4
| ├─ ...
├─ reconstruction_input
| ├─ dan-X01
| | ├─ dan-X01 (video frames)
| | ├─ dan-X01_expose
| | ├─ dan-X01_parsing
| | ├─ dan-X01_pifu
| | ├─ dan-X01_smpl
| ├─ dan-X02
| | ├─ ...
| ├─ ...
├─ reconstruction_output (reconstructed garment meshes)
| ├─ dan-X01
| ├─ dan-X02
| ├─ ...
├─ regressor_training_data
├─ train_sequences
| ├─ meshes (reconstructed garment meshes in Tpose)
| | ├─ dan-X01
| | ├─ dan-X02
| | ├─ ...
| ├─ poses (encoded poses using the SoftSMPL encoding)
| | ├─ dan-X01
| | ├─ dan-X02
| | ├─ ...
├─ validation_sequences (same structure as train)
├─ ...
Datasets for the reconstruction script are made by processing each frame with:
- ExPose (output is SMPL-X, they need to be converted to SMPL too)
- PifuHD
- Self-Correction-Human-Parsing
The necessary files are provided in the reconstruction_input folder. We also provide reconstructed meshes for each dataset (reconstruction_input folder) and the same meshes in Tpose space (inside the meshes folder on regressor_training_data).
Our regressors predict wrinkles (vertex displacements with respect to a template mesh) from SMPL poses encoding using the SoftSMPL encoding. We provide such encoded poses for the DataDanGrey dataset and also the scripts to generate such encoding from arbitrary SMPL paramteres.
You can use AMASS sequences by placing the .npz files under data/test_sequence.
Alternatively, you can run the regression on sequences of SMPL poses saved as .pkl files. Check the set
of reconstructed scripts.
@article {casado2022pergamo,
journal = {Computer Graphics Forum (Proc. of SCA), 2022},
title = {{PERGAMO}: Personalized 3D Garments from Monocular video},
author = {Casado-Elvira, Andrés and Comino Trinidad, Marc and Casas, Dan},
year = {2022}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent