chongjiange / metabev Goto Github PK

• (20/04/2023) MetaBEV is released on arxiv.

Perception systems in modern autonomous driving vehicles typically take inputs from complementary multi-modal sensors, e.g., LiDAR and cameras. However, in real-world applications, sensor corruptions and failures lead to inferior performances, thus compromising autonomous safety.

In this paper, we propose a robust framework, called MetaBEV, to address extreme real-world environments, involving overall six sensor corruptions and two extreme sensor-missing situations.

Experiments show MetaBEV outperforms prior arts by a large margin on both full and corrupted modalities. For instance, when the LiDAR signal is missing, MetaBEV improves 35.5% detection NDS and 17.7% segmentation mIoU upon the vanilla BEVFusion model; and when the camera signal is absent, MetaBEV still achieves 69.2% NDS and 53.7% mIoU, which is even higher than previous works that perform on full-modalities. Moreover, MetaBEV performs fairly against previous methods in both canonical perception and multi-task learning settings, refreshing state-of-the-art nuScenes BEV map segmentation with 70.4% mIoU.

Our model achieves the following performance on :

• Detection on nuScenes val set with LiDAR and Camera.

• Segmentation on nuScenes val set with LiDAR and Camera.

The project is based on mmdetection3d, BEVFusion, robust benchmark. Thanks for their awesome works.

This project is under the MIT license. See LICENSE for details.

If you find MetaBEV useful or relevant in your research please consider citing our paper:

@article{ge2023metabev,
  title={MetaBEV: Solving Sensor Failures for BEV Detection and Map Segmentation},
  author={Ge, Chongjian and Chen, Junsong and Xie, Enze and Wang, Zhongdao and Hong, Lanqing and Lu, Huchuan and Li, Zhenguo and Luo, Ping},
  journal={arXiv preprint arXiv:2304.09801},
  year={2023}
}