runa91 / barc_release Goto Github PK

View Code? Open in Web Editor NEW

69.0 5.0 9.0 5.17 MB

barc_release

License: Other

Python 100.00%

barc_release's Introduction

BARC

[Project Page]

Description
Installation
Usage
- Demo
- Training
- Inference
Citation
License
Acknowledgments
Contact

Description

Breed Augmented Regression using Classification (BARC) is a method for dog pose and shape estimation.

Installation

Environment

The code has been tested with CUDA 10.1, CuDNN 7.5, Python 3.7 and PyTorch 1.6.0.

    conda create -n "conda_barc" python=3.7.6 ipython
    conda activate conda_barc
    conda install pytorch==1.6.0 torchvision cudatoolkit=10.1 -c pytorch

To install the remaining dependencies run the following command:

    pip install -r requirements.txt

Data Preparation

All necessary data be downloaded here. A folder named 'checkpoint' contains pretrained models, copy it to the main folder of this project. A folder named 'stanext_related_data.zip' contains information related to the dataset. Please extract the zip file and copy it to data/stanext_related_data.

Download the Stanford Extra image dataset from https://github.com/benjiebob/StanfordExtra and store it in datasets/StanfordExtra_V12.

Your folder structure should look as follows:

folder
├── checkpoint
│   ├── barc_complete
│   └── ...
├── data
│   ├── breed_data
│   ├── smal_data
│   ├── statistics
│   └── stanext_related_data
├── datasets
│   ├── test_image_crops
│   ├── StanfordExtra_V12
│   │   ├── StanExtV12_Images
│   │   └── labels
├── scripts
│   └── ...
├── src
│   └── ...

Configurations

All configuration files can be found in src/configs. You will need to adjust paths in barc_cfg_{train,test,visualization}.yaml and dataset_path_configs.py. If desired you can change the weights for different loss functions used at training time, see barc_loss_weights.json. We do not recommend changing zero-value weights to non-zero values, as most of the unused loss functions were removed.

Usage

Demo

In order to run our pretrained model on new sample images, prepare image crops and put them into the folder datasets/test_image_crops. The crops can have arbitrary rectangular shape, but should show the dog more or less in the center of the image. Please have a look at the provided example image.

    python scripts/visualize.py --workers 12  \
    --model-file-complete barc_complete/model_best.pth.tar \
    --config barc_cfg_visualization.yaml \

Training

You can train a full new model using the following command. Pretrained weights for the stacked hourglass and the 3d pose branch (including the normalizing flow pose prior) will be loaded.

    python scripts/train.py --workers 12 --checkpoint barc_new_v2 \
    --config barc_cfg_train.yaml \
    start \
    --model-file-hg barc_hg_pret/checkpoint.pth.tar \
    --model-file-3d barc_normflow_pret/checkpoint.pth.tar

Inference

In order to reproduce the results as listed in our paper, run the following command:

    python scripts/test.py --workers 12  \
    --model-file-complete barc_complete/model_best.pth.tar \
    --config barc_cfg_test.yaml
    --save-images True

Citation

If you find this Model & Software useful in your research we would kindly ask you to cite:

@inproceedings{BARC:2022,
    title = {BARC: Learning to Regress 3D Dog Shape from Images by Exploiting Breed Information},
    author = {Rueegg, Nadine and Zuffi, Silvia and Schindler, Konrad and Black, Michael J.},
    booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
    year = {2022}
}

License

Software Copyright License for non-commercial scientific research purposes. Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use BARC data, model and software, (the "Data & Software"), including 3D meshes, images, videos, textures, software, scripts, and animations. By downloading and/or using the Data & Software (including downloading, cloning, installing, and any other use of the corresponding github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Data & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License.

Acknowledgments

Contact

The code of this repository was implemented by Nadine Rüegg.

For commercial licensing (and all related questions for business applications), please contact [email protected].

barc_release's People

Contributors

Stargazers

Watchers

Forkers

pixelknit bobby20180331 mshooter repo-collection yukunchen jackzhousz jmcoholich easychart timenolong

barc_release's Issues

How to train segmentation through an hourglass network?

Thank you for your work！

I see you added this paragraph to the hourglass network code, and want to ask if you can provide your complete training process code

This code only trains keypoints：https://github.com/anibali/pytorch-stacked-hourglass

Why do you choose the latter two heatmap? If you train 24 keypoints, do you need to set NUM_JOINT to 26?

barc_hg_pret.pth checkpoint training code share.

@runa91 Thanks a lot for your shares.
Barc result is really realistic, compared with SOTA method before. However, the speed seems slower than our expectation.So could you share the Hourglass training code used for the barc_hg_pret.pth.tar checkpoint that provide in the code. Thus we can evaluate the hg8/hg2/hg1 performance. Thanks.

3D image grid, train your own image

I see an application on the official website that can upload pictures and download 3D grids, but it is no longer available. How can I get the 3D images I want to train for my own models

No such file or directory : `data/smal_samples/dogs_set_torch_1/statistics_modified_v1.json`

Hi, when running the demo command I get the following stack :

Traceback (most recent call last):
  File "scripts/visualize.py", line 24, in <module>
    from stacked_hourglass.datasets.imgcrops import ImgCrops
  File "scripts/../src/stacked_hourglass/datasets/imgcrops.py", line 11, in <module>
    from configs.anipose_data_info import COMPLETE_DATA_INFO        
  File "scripts/../src/stacked_hourglass/datasets/../../configs/anipose_data_info.py", line 42, in <module>
    STATISTICS = load_statistics(statistics_path)
  File "scripts/../src/stacked_hourglass/datasets/../../configs/anipose_data_info.py", line 27, in load_statistics
    with open(statistics_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/ps/scratch/nrueegg/new_projects/Animals/data/smal_samples/dogs_set_torch_1/statistics_modified_v1.json'

I downloaded the data from the provided link, however there is no smal_sample directory neither in it, nor in the repository's data/ directory. Is this a file that we are supposed to obtain from somewhere else ?

No module named 'posemapper'

run the inference script, and the error

  File "scripts/test.py", line 23, in <module>
    from combined_model.model_shape_v7 import ModelImageTo3d_withshape_withproj
  File "scripts/../src/stacked_hourglass/datasets/../../combined_model/model_shape_v7.py", line 19, in <module>
    from smal_pytorch.smal_model.smal_torch_new import SMAL
  File "scripts/../src/stacked_hourglass/datasets/../../combined_model/../lifting_to_3d/../smal_pytorch/smal_model/smal_torch_new.py", line 46, in <module>
    from .smal_basics import align_smal_template_to_symmetry_axis, get_dog_template, get_neutral_template, get_symmetry_indices
  File "scripts/../src/stacked_hourglass/datasets/../../combined_model/../lifting_to_3d/../smal_pytorch/smal_model/smal_basics.py", line 12, in <module>
    from smpl_webuser.serialization import load_model
  File "scripts/../src/stacked_hourglass/datasets/../../combined_model/../lifting_to_3d/../smpl_webuser/serialization.py", line 30, in <module>
    from posemapper import posemap
ModuleNotFoundError: No module named 'posemapper'

I have extracted the smpl_webuser module from SMPL_python_v.1.0.0.zip, and put it in the path barc_root/src, whick is like the figure show as below

is the smpl_webuser path error? how can I fix this?

time consuming and low gpu utilization for training the model

24 batch size for stanford training data, the gpu memory is about 13GB, the gpu utilization is not stable from 0% to 90%, and most of time is 0% waiting for data. does the 3d process resnet, hg8 and model3d entirely compute on gpu?

Canonical T-pose mesh?

Many thanks for your amazing job!

May I ask how to generate the canonical T-pose mesh in Figure 1 in your paper?

wrong tensor size

/barc_release/scripts/../src/combined_model/model_shape_v7.py", line 419, in forward
pred_pose_rot6d = output['pose'] + pose_rot6d_mean_zeros
RuntimeError: The size of tensor a (35) must match the size of tensor b (12) at non-singleton dimension 1

No such file or directory: StanfordExtra_V12/labels/train_stanford_StanfordExtra_v12.npy

hi, thansk for great works!
when i trainning model by following commands

    python scripts/train.py --workers 12 --checkpoint barc_new_v2 \
    --config barc_cfg_train.yaml \
    start \
    --model-file-hg barc_hg_pret/checkpoint.pth.tar \
    --model-file-3d barc_normflow_pret/checkpoint.pth.tar

it failed, error details :

FileNotFoundError: [Errno 2] No such file or directory: xx/StanfordExtra_V12/labels/train_stanford_StanfordExtra_v12.npy,

if some files are missing, when i download StanfordExtra_V12 datasets, i can't find any train_stanford_StanfordExtra_v12.npy file in directory of StanfordExtra_V12/labels

No module named 'stacked_hourglass.datasets'

Hi, I have installed the package pytorch-stacked-hourglas, and the version is 1.0.0a9.
but the error No module named 'stacked_hourglass.datasets' occurred , when I run the inference script
python scripts/test.py --workers 12 --model-file-complete barc_complete/model_best.pth.tar --config barc_cfg_test.yaml.

btw, It seems that the config params of inference and demo are inverse?

train.py use too much GPU memory.

RuntimeError: CUDA out of memory. Tried to allocate 192.00 MiB (GPU 0; 31.75 GiB total capacity; 30.38 GiB already allocated; 85.50 MiB free; 30.63 GiB reserved in total by PyTorch)

when I use the training code .32GiB GPU memory is not enough.