This repo is official PyTorch implementation of Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wild (CVPR 2023).

1. Slightly change torchgeometry kernel code following here.
2. Prepare human_model_files folder following below Directory part and place it at common/utils/human_model_files.
3. Move to demo folder.
4. Download pre-trained InterWild from here.
5. Put input images at images. The image should be a cropped image, which contain a single human. For example, using a human detector.
6. Run python demo.py --gpu $GPU_ID
7. Boxes, meshes, and MANO parameters are saved at boxes, meshes, params, respectively.

The ${ROOT} is described as below.

${ROOT}
|-- data
|-- demo
|-- common
|-- main
|-- output

• data contains data loading codes and soft links to images and annotations directories.
• demo contains the demo code
• common contains kernel code. You should put MANO_RIGHT.pkl and MANO_LEFT.pkl at common/utils/human_model_files/mano, where those are available in here.
• main contains high-level codes for training or testing the network.
• output contains log, trained models, visualized outputs, and test result.

You need to follow directory structure of the data as below.

${ROOT}
|-- data
|   |-- InterHand26M
|   |   |-- annotations
|   |   |   |-- train
|   |   |   |-- test
|   |   |-- images
|   |-- MSCOCO
|   |   |-- annotations
|   |   |   |-- coco_wholebody_train_v1.0.json
|   |   |   |-- coco_wholebody_val_v1.0.json
|   |   |   |-- MSCOCO_train_MANO_NeuralAnnot.json
|   |   |-- images
|   |   |   |-- train2017
|   |   |   |-- val2017
|   |-- HIC
|   |   |-- data
|   |   |   |-- HIC.json

• Download InterHand2.6M [HOMEPAGE]. images contains images in 5 fps, and annotations contains the H+M subset.
• Download the whole-body version of MSCOCO [HOMEPAGE]. MSCOCO_train_MANO_NeuralAnnot.json can be downloaded from [here].
• Download HIC [HOMEPAGE] [annotations]. You need to download 1) all Hand-Hand Interaction sequences (01.zip-14.zip) and 2) some of Hand-Object Interaction seuqneces (15.zip-21.zip) and 3) MANO fits. Or you can simply run python download.py in the data/HIC folder.
• All annotation files follow MSCOCO format.
• If you want to add your own dataset, you have to convert it to MSCOCO format.

You need to follow the directory structure of the output folder as below.

${ROOT}
|-- output
|   |-- log
|   |-- model_dump
|   |-- result
|   |-- vis

• log folder contains training log file.
• model_dump folder contains saved checkpoints for each epoch.
• result folder contains final estimation files generated in the testing stage.
• vis folder contains visualized results.

• As InterHand2.6M is too large, I only use data whose ann_id belong to the H subset, while values belong to the H+M subset.
• To this end, please download aid_human_annot_train.txt and aid_human_annot_test.txt from here and here, respectively, and place them in main folder.
• Slightly change torchgeometry kernel code following here.
• Prepare human_model_files folder following above Directory part and place it at common/utils/human_model_files.

In the main folder, run

python train.py --gpu 0-3

to train the network on the GPU 0,1,2,3. --gpu 0,1,2,3 can be used instead of --gpu 0-3. If you want to continue experiment, run use --continue.

• If you want to test with pre-trained InterWild, download it from here and place it at `output/model_dump'.
• Or if you want to test with our own trained model, place your model at output/model_dump.

In the main folder, run

python test.py --gpu 0-3 --test_epoch 6

to test the network on the GPU 0,1,2,3 with snapshot_6.pth.tar. --gpu 0,1,2,3 can be used instead of --gpu 0-3.

@InProceedings{Moon_2023_CVPR_InterWild,  
author = {Moon, Gyeongsik},  
title = {Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wild},  
booktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},  
year = {2023}  
}  

This repo is CC-BY-NC 4.0 licensed, as found in the LICENSE file.

[Terms of Use] [Privacy Policy]