: We propose a framework for efficient active learning within various 3D object detection techniques and modalities, demonstrating the effectiveness of active learning at reaching comparable detection performance on benchmark datasets at a fraction of the annotation cost. Datasets include roadside infrastructure sensors (top row) and onboard vehicle sensors (bottom row), with LiDAR-only and LiDAR+camera fusion methods, the two dominant strategies in state-of-the-art performance at the safety-critical detection task.

This is the official implementation of our paper:

[IV 2024] ActiveAnno3D - An Active Learning Framework for Multi-Modal 3D Object Detection [arXiV] [website]

ActiveAnno3D is the first active learning framework for multi-modal 3D object detection. With this framework you can select data samples for labeling that are of maximum informativeness for training.

In summary:

1. We explore various continuous training methods and integrate the most efficient method regarding computational demand and detection performance.
2. We perform extensive experiments and ablation studies with BEVFusion and PV-RCNN on the nuScenes and TUM Traffic Intersection dataset.
3. We show that we can achieve almost the same performance with PV-RCNN and the entropy-based query strategy when using only half of the training data (77.25 mAP compared to 83.50 mAP) of the TUM Traffic Intersection dataset.
4. BEVFusion achieved an mAP of 64.31 when using half of the training data and 75.0 mAP when using the complete nuScenes dataset.
5. We integrate our active learning framework into the proAnno labeling tool to enable AI-assisted data selection and labeling and minimize the labeling costs.

The generalized active learning flow involves the selection of data from an unlabeled pool according to an acquisition function, which, in the case of uncertainty-driven AL, utilizes the trained model or, in the case of diversity-driven AL, may be independent of the training. This selected data is then annotated by an oracle and aggregated with previously labeled data. Whether or not all data or just the new data is used in the next training step is determined by the choice of training strategy. The variety of possible acquisition and training techniques and unique domain challenges posed by autonomous driving make active learning an opportune environment for innovation toward safe and accurate learning.

For installation using Docker, please refer to the INSTALL.md file.

The code is tested in the following python environment:

• Python 3.7.16
• PyTorch 1.10.1
• CUDA 11.3.1
• spconv-cud113=2.21.12
• Open3D 0.16.0
• Wandb 0.16.2

1. Incorporate Continuous Training Strategies [link]
2. Propose Temporal CRB [link]
3. Propose Class-weighted CRB [link]
4. Post process Predictions [link]
5. Develop develop an interface for the proAnno labeling tool [link]

The dataset, e.g. TUM Traffic Intersection, has to be in the KITTI format. If it's in OpenLABEL format then use our converter

python tumtraf_converter.py --load_dir /home/user/tumtraf --save_dir /home/user/tumtraf_kitti_format --splits train val test

To run normal training:

in ./tools/cfgs add a folder <dataset_models> then add the .yaml configuration files for your models there.

python ./tools/main/train.py --cfg_file <path-to-yaml-file> --extra_tag normalTraining

if you want to use a pre-trained model set

--pretrained_model to <path-to-your-pretrained-model>

To run active training:

in ./tools/cfgs add a folder <active-dataset_models> then add the .yaml configuration files for your models there.

python ./tools/main/train.py --cfg_file <path-to-yaml-file> --extra_tag <continuous_training_method>

1. The flask_app.py file should be located anywhere outside the docker container.

2. To access this flask server from the proAnno tool running on a different machine, we should use the IP address of the machine where the flask app is running, followed by the port number.

   for example, if the IP address of the workstation is 192.168.1.5 and the flask server is running on port 5000, then from the proAnno tool on another machine we would access the flask app using 'http://192.168.1.5:5000'

3. Run flask_app.py

4. ActiveAnno3D is ready for the annotator to send a command.

5. The selected point cloud frames will be saved in ActiveAnno3D under: ./data/proannoV2/currently_annotating, and they have to be copied manually to the proAnno side for annotations.

The graph illustrates the mAP score achieved by the BEVFusion model on the nuScenes dataset relative to the expanding size of the training set in the active learning setting with random and entropy queries separately.

The graph illustrates the mAP scores achieved by the PV-RCNN model on the TUM Traffic Intersection dataset relative to the expanding size of the training set in the active learning setting with random and entropy queries separately. The graph illustrates the mAP scores achieved by the PV-RCNN model on the TUM Traffic Intersection dataset relative to the expanding size of the training set in the active learning setting with eight different query strategies.

Qualitative results are illustrated by two pairs of images. The left pair is from the TUM Traffic Intersection dataset, and the right pair is from nuScenes. For each pair, the left image shows the predicted labels for each class, with each class represented by a different color. The right image of each pair shows the predictions made by learning on the complete dataset. Both results are quite similar, showing the efficiency of the active learning technique.

Our baseline is the Active3D framework Active3D, an active learning framework for 3D object detection, proposing the CRB query strategy that assesses the point cloud data informativeness based on 3 data characteristics: 3D box class distribution, feature representativeness, and point density distribution.

If you find our work useful in your research, please cite our work and ⭐ our repository.

@misc{activeanno3d,
      title={ActiveAnno3D - An Active Learning Framework for Multi-Modal 3D Object Detection}, 
      author={Ghita, Ahmed and Antoniussen, Bjørk and Zimmer,Walter and Greer,Ross and Creß, Christian and Møgelmose,Andreas and Trivedi, Mohan M. and Knoll,Alois C.},
      year={2024},
      eprint={2402.03235},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

The ActiveAnno3D framework is licensed under CC BY-NC-SA 4.0.