This repository contains an implementation of a convolution neural network for 3D object detection in the fusion of lidar point clouds and camera images. As shown in the figure below, the model architecture consists of three major components: frustum-aware generator, decorator, and fusion-based detection network.

_Note: It is built upon pytorch-geometric and provides usage with the KITTI and nuScenes dataset.

Results of our FADN model for 3D object detection on both the KITTI and the nuScenes dataset.

• OS: Ubuntu 20.04 LTS
• CUDA: 11.3
• cuDNN: 8
• pytorch 1.10.0

Inside the project folder create a "FADN/data" folder and within this folder, create a "data/output" subfolder. The trained models and evaluations will be stored in that folder. Depending on the desired dataset, create the following additional subfolders inside the "data" folder:

datasets/kitti/
datasets/nuscenes/

In a second step follow the instructions of the KITTI and nuScenes websites to download and store the datasets in the created subfolders.

Finally, clone this repository into the project folder using the command:

git clone https://github.com/denyz/FADN.git

|  
+---FADN/  
|   |  
|   +---data/  
|   |   |  
|   |   +---kitti/  
|   |   |   |
|   |   |   +---ImageSets/
|   |   |   |	|   +---train.txt
|   |   |   |	|   +---val.txt
|   |   |   |	|   +---test.txt
|   |   |   +---gt_database/
|   |   |   +---trainning/
|   |   |   |	|   +---calib/
|   |   |   |   |   +---image_2
|   |   |   |   |   +---image_3
|   |   |   |   |   +---label_2
|   |   |   |   |   +---planes
|   |   |   |	|   +---velodyne
|   |   |   |	|   +---decorated_lidar
|   |   |   +---testing/
|   |   |   |   |   +---calib/
|   |   |   |   |   +---image_2
|   |   |   |   |   +---image_3
|   |   |   |   |   +---planes
|   |   |   |   |   +---velodyne
|   |   |   |
|   |   |   +---kitti_infos_test.pkl
|   |   |   +---kitti_infos_train.pkl
|   |   |   +---kitti_infos_trainval.pkl
|   |   |
|   +---tools/  
| 
.
.
.
+---...

FADN is a LiDAR-camera fusion 3D detection framework. It supports many popular datasets like KITTI and nuscenes. To install the FADN please first install its requirements.txt. And as we modify some parts of the OpenPCDet LIB to support the decorated KITTI dataset. We packaged the development environment into docker, which will be released later. To install it, run the following commands.

$ python setup.py develop

The overall pipeline is divided into three major steps.

• Creation of a decorated dataset from the raw KITTI or nuScenes dataset
• Creation and training of a model based on the created decorated dataset
• Evaluation of the trained model

The settings of all three steps are defined in a unified configuration file, which must consequently be created first.

the decorated dataset needs to be created by converting the lidar point clouds of the raw datasets to a decorated data structure. This will generate the decorated_lidar folder in the dataset. To do this, execute the following command inside the docker container:

$ cd FADN
$ python tools/decorating.py

usage:          decorating.py [--dataset] [--config]

arguments:
    --dataset   Path to the raw (RadarScenes/nuScenes) dataset.
    --config  Parameters to the created decorated dataset.

Create the KITTI PKL

python -m pcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/decorate_kitti_dataset.yaml

Next step, you can use the created decorated dataset to train a model. To do this, run the following command:

$ python -m pcdet.datasets.kitti.decorate_kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/decorate_kitti_dataset.yaml
$ cd tools

usage:   train.py [--data] [--results] [--config]
$ python tools/train.py --cfg_file cfgs/kitti_models/FADN_decorated.yaml

Finally, you can evaluate a trained model using the following command in kittiEval:

usage:   ./eval_detection_3d_offline [gt_dir] [result_dir]

The evaluation metrics include :

• Overlap on image (AP)
• Oriented overlap on image (AOS)
• Overlap on ground-plane (AP)
• Overlap in 3D (AP)

Within the provided "results" folder a new "data" and "plot" folder is created, in which the evaluation results are saved.