provide comprehensive information for nuInsSeg dataset

Through investigation, we found that although existing large-scale autonomous driving datasets provide multi-modal data, they lack multi-modal and consistent annotations. So on the basis of the nuScenes dataset, we supplement the image instance segmentation annotation, and call it nuInsSeg. As far as we know, this is the first one that contains both: LiDAR point cloud, RGB image, Open source autonomous driving datasets with manual 2D annotations (2D detection boxes, 2D instance segmentation), 3D annotations (3D detection boxes...), and these 2D and 3D annotations have instance-level consistency. Added 947,072 2D annotations (box detection and instance segmentation) to the 204,894 images in the nuScenes-trainval data.

It is hoped that nuInsSeg can help researchers to better study large-scale autonomous driving scenarios, especially from the perspective of multimodal sensor fusion.

The nuScenes training and validation dataset contains 1,166,187 3D detection box annotations, but because many objects are occluded or unclear in the camera's line of sight, the corresponding 2D annotations will be less. The following are the data statistics of the nuInsSeg data training set and validation set, and the target category is consistent with the nuScenes detection task.

Here we provide the benchmark for instance segmentation and Weakly supervised instance segmentation on nuInsSeg dataset. The category of instance segmentation is the same as nuScenes detection tasks.

We adopt a standard evaluation metric of instance segmentation which is the same as that used in Mask R-CNN. It includes AP (average precision over IoU thresholds), AP50, AP75 and APs, APm, APl (AP at different scale). In our experiment, models are implemented using the official codebase without modification.

Instance Segmentation aims to recognize distinct instances of objects in an image by predicting pixel-level category and instance identity, which has benefited a wide range of applications such as autonomous driving. Here we provide the results of several compatitable models on nuInsSeg dataset(official code).

R-50 and R-101 is pretrained on imagenet. 'Lr_schedule' means the training iterations and '1x' is 7w iterations.

Weakly supervised instance segmentation aims to extract objects with simple and cheap annotation such as image-level tags, scribble and bounding box instead of expensive pixel-level annotations.

Here we explain different annotations used in the exp. 'box' means only using the 2D bounding box annotation for each instance, 'point' means using a specific number of points with human annotation indicating the background/foreground, '3dbox' means using the 3D bounding box annotations for each instance and 'pc' means the original point cloud.

LWSIS is the method used in our paper "LWSIS: LiDAR-guided Weakly Supervised Instance Segmentation for Autonomous Driving."

We provide development tookit to access the instance segmentation results easily: nuInsSeg-devkit.

Here we provide the download link of the nuInsSeg dataset.

1. nuInsSeg

2. nuInsSeg-mini

It is stated here that we only add the annotation of instance segmentation on the basis of the nuscenes dataset, in order to better conduct research on multimodal perception for researchers. Many thanks to nuscnes for contributions to open source data.