The goal of this project is to use diffusion model to perform affordance segmentation.
- Run generate superpoint
python dps/scripts/gen_superpoint.py
- Run preprocess
python preprocess_data.py
- Set environment variable
RPDIFF_SOURCE_DIR
:
export RPDIFF_SOURCE_DIR=$PWD/dps/external/rpdiff
- Run meshcat viewer:
meshcat-server
We only tested on CUDA11.8
.
- Install other dependency
pip install -r requirements.txt
- Install torch 2.1.0 first. This is to get adapt with torch3d.
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118
- Install pytorch3d's dependency:
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
- Install pytorch3d from source
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
- Install torch_scatter.
pip install torch_geometric
pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.1.0+cu118.html
- Install detectron2
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
- Build code for superoint; Comment out everything before
echo "โญ Installing FRNN"
You need to replace the line 19: extra_compile_args = {"cxx": ["-std=c++14"]}
to extra_compile_args = {"cxx": ["-std=c++17"]}
in external/superpoint_transformer/src/dependencies/FRNN/setup.py
.
cd external/superpoint_transformer
./install.sh
You need to install pytorch3d
, superpoint
.
- [Superpoint] (https://github.com/drprojects/superpoint_transformer)
- Install detectron2.
- Fix the normal for shape-completion.
- Link the system with rpdiff pipeline.
- Add training for real object.
- Improving the bbox points.
- We can augment the pose registration with superpoint-augmentation; randomly dropping or superpoint.
- We can also apply a range filter.
- Fix the segmentation fault. (I guess the output pose is wrong or something like that)