This is a Pytorch-Lightning implementation of the paper "Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks".
Given a sequence of P past point clouds (left in red) at time T, the goal is to predict the F future scans (right in blue).
Overview of our architecture
If you use our code in your academic work, please cite the corresponding paper:
@inproceedings{mersch2021corl,
author = {B. Mersch and X. Chen and J. Behley and C. Stachniss},
title = {{Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks}},
booktitle = {Proc.~of the Conf.~on Robot Learning (CoRL)},
year = {2021},
}Download the Kitti Odometry data from the official website.
Clone this repository and run
cd point-cloud-prediction
git submodule update --initto install the Chamfer distance submodule. The Chamfer distance submodule is originally taken from here with some modifications to use it as a submodule. All parameters are stored in config/parameters.yaml.
In this project, we use CUDA 10.2. All other dependencies are managed with Python Poetry and can be found in the poetry.lock file. If you want to use Python Poetry (recommended), install it with:
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -Install Python dependencies with Python Poetry
poetry installand activate the virtual environment in the shell with
poetry shellWe process the data in advance to speed up training. The preprocessing is automatically done if GENERATE_FILES is set to true in config/parameters.yaml. The environment variable PCF_DATA_RAW points to the directory containing the train/val/test sequences specified in the config file. It can be set with
export PCF_DATA_RAW=/path/to/kitti-odometry/dataset/sequencesand the destination of the processed files PCF_DATA_PROCESSED is set with
export PCF_DATA_PROCESSED=/desired/path/to/processed/data/Note If you have not pre-processed the data yet, you need to set GENERATE_FILES: True in config/parameters.yaml. After that, you can set GENERATE_FILES: False to skip this step.
The training script can be run by
python pcf/train.pyusing the parameters defined in config/parameters.yaml. Pass the flag --help if you want to see more options like resuming from a checkpoint or initializing the weights from a pre-trained model. A directory will be created in pcf/runs which makes it easier to discriminate between different runs and to avoid overwriting existing logs. The script saves everything like the used config, logs and checkpoints into a path pcf/runs/COMMIT/EXPERIMENT_DATE_TIME consisting of the current git commit ID (this allows you to checkout at the last git commit used for training), the specified experiment ID (pcf by default) and the date and time.
Example:
pcf/runs/7f1f6d4/pcf_20211106_140014
7f1f6d4: Git commit ID
pcf_20211106_140014: Experiment ID, date and time
Test your model by running
python pcf/test.py -m COMMIT/EXPERIMENT_DATE_TIMEwhere COMMIT/EXPERIMENT_DATE_TIME is the relative path to your model in pcf/runs. Note: Use the flag -s if you want to save the predicted point clouds for visualiztion and -l if you want to test the model on a smaller amount of data.
Example
python pcf/test.py -m 7f1f6d4/pcf_20211106_140014or
python pcf/test.py -m 7f1f6d4/pcf_20211106_140014 -l 5 -sif you want to test the model on 5 batches and save the resulting point clouds.
After passing the -s flag to the testing script, the predicted range images will be saved as .svg files in pcf/runs/COMMIT/EXPERIMENT_DATE_TIME/range_view_predictions. The predicted point clouds are saved to pcf/runs/COMMIT/EXPERIMENT_DATE_TIME/test/point_clouds. You can visualize them by running
python pcf/visualize.py -p pcf/runs/COMMIT/EXPERIMENT_DATE_TIME/test/point_clouds
Five past and five future ground truth and our five predicted future range images.
Last received point cloud at time T and the predicted next 5 future point clouds. Ground truth points
are shown in red and predicted points in blue.
You can download our best performing model from the paper here. Just extract the zip file into pcf/runs.
This project is free software made available under the MIT License. For details see the LICENSE file.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent