- Linux or macOS (Tested on Ubuntu 20.04)
- Python 3
- CPU or NVIDIA GPU + CUDA CuDNN
- Conda
# clone the repo
git clone https://github.com/hshi74/robocraft3d.git
# create the conda environment
conda env create -f robocraft.yml
conda activate robocraft
# install requirements for the simulator
pip install -e simulator
# install pytorch
pip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113
# add this line to your bashrc
export PYTHONPATH="${PYTHONPATH}:/path/to/robocraft3d"
See the controller codebase of RoboCraft
For all the following bash or python scripts, you will need to modify certain hyperparameters (like directories) before you run them.
bash perception/scripts/run_sample.sh
.- (Optional) There could be a tiny portion of problematic datapoints. If you want to manually check the sampling results and remove the problematic ones,
- Run
perception/scripts/inspect_perception.sh
to move all the visualizations into the same folder for your convenience - Manually go through all the videos and type the indices of problematic videos into
dump/perception/inspect/inspect.txt
- Run
perception/scripts/clean_perception.sh
to remove all the problemtaic ones.
- Run
- Run
percetion/scripts/make_dataset.py
to build the dataset for the GNN
bash dynamics/scripts/run_train.sh
bash planning/scripts/run_control.sh
config/
: config files for perception, dyanmics, planning, and simulationdynamics/
: scripts to train and evaluate the GNN.geometries/
: the STL files for tools and assets and surface point cloud representations for tools (in the.npy
files)models/
: a GNN checkpoint trained by us and its configurations (in the.npy
file)perception/
: the perception module of RoboCraftplanning/
: the planning module of RoboCraftsimulator/
: the simulation environment PlasticineLabtarget_shapes/
: point clouds of some target shapesutils/
: utility and visualization functions
If you use the codebase in your research, please cite:
@article{doi:10.1177/02783649231219020,
author={Haochen Shi and Huazhe Xu and Zhiao Huang and Yunzhu Li and Jiajun Wu},
title={RoboCraft: Learning to see, simulate, and shape elasto-plastic objects in 3D with graph networks},
journal={The International Journal of Robotics Research},
volume={0},
number={0},
pages={02783649231219020},
year={2023},
doi={10.1177/02783649231219020},
URL={https://doi.org/10.1177/02783649231219020}
}