This is the official implementation of VPFNet: Improving 3D Object Detection with Virtual Point based LiDAR and Stereo Data Fusion, built on OpenPCDet.

@ARTICLE{VPFNet,
    author={Zhu, Hanqi and Deng, Jiajun and Zhang,
    Yu and Ji, Jianmin and Mao, Qiuyu and Li, Houqiang
    and Zhang, Yanyong},
    journal={IEEE Transactions on Multimedia},
    title={VPFNet: Improving 3D Object Detection
    with Virtual Point based LiDAR and Stereo Data
    Fusion},
    year={2022}
}

1. Prepare for the running environment.

   For snake, follow the installation steps in snake. For VPFNet, use the docker image provided here, or follow the installation steps in OpenPCDet.

   docker pull djiajun1206/pcdet-pytorch1.5
   

2. Prepare for the data.

   Please download the Kins dataset and official KITTI 3D object detection dataset. Train the snake model using the split in /data/kitti/ImageSets/train.txt, then generate the 2D bounding box and 2D instance mask.

   python run.py --type evaluate --cfg_file configs/kins_snake.yaml test.dataset KinsVal     
   

   Then organize the files as follows (the road planes could be downloaded from [road plane]:

   VPFNet
   ├── data
   │   ├── kitti
   │   │   │── ImageSets
   │   │   │── training
   │   │   │   ├──calib & velodyne & label_2 & image_2 & image_3 &  planes & image_2_box & image_2_snake & image_3_box & image_3_snake  
   │   │   │── testing
   │   │   │   ├──calib & velodyne & image_2 & image_3
   ├── pcdet
   ├── tools
   

   Generate the data infos by running the following command:

   python -m pcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml
   

3. Setup.

   python setup.py develop
   

1. Training.

   The configuration file is in tools/cfgs/kittimodels, and the training scripts is in tools/scripts.

   cd tools
   bash ./scripts/dist_train.sh 8 --cfg_file  ./cfgs/kitti_models/VPFNet.yaml  --batch_size 16 --epochs 80 --workers 8
   

2. Evaluation.

   The configuration file is in tools/cfgs/VPFNet, and the training scripts is in tools/scripts.

   cd tools
   python test.py --cfg_file  ./cfgs/kitti_models/VPFNet.yaml  --batch_size 4  --eval_all
   

Thanks to the strong and flexible OpenPCDet codebase maintained by Shaoshuai Shi (@sshaoshuai) and Chaoxu Guo (@Gus-Guo).

This repository is implemented by Hanqi Zhu ([email protected]).