xinge008 / cylinder3d Goto Github PK

Rank 1st in the leaderboard of SemanticKITTI semantic segmentation (both single-scan and multi-scan) (Nov. 2020) (CVPR2021 Oral)

License: Apache License 2.0

Python 97.77% Shell 0.30% Dockerfile 1.93%

point-cloud lidar-point-cloud panoptic-segmentation semantickitti nuscenes lidar-segmentation semantic-segmentation cvpr2021

cylinder3d's Introduction

Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation

The source code of our work "Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation

News

2022-06 [NEW:fire:] PVKD (CVPR2022), a lightweight Cylinder3D model with much higher performance has been released here
Cylinder3D is accepted to CVPR 2021 as an Oral presentation
Cylinder3D achieves the 1st place in the leaderboard of SemanticKITTI multiscan semantic segmentation

Cylinder3D achieves the 2nd place in the challenge of nuScenes LiDAR segmentation, with mIoU=0.779, fwIoU=0.899 and FPS=10Hz.
2020-12 We release the new version of Cylinder3D with nuScenes dataset support.
2020-11 We preliminarily release the Cylinder3D--v0.1, supporting the LiDAR semantic segmentation on SemanticKITTI and nuScenes.
2020-11 Our work achieves the 1st place in the leaderboard of SemanticKITTI semantic segmentation (until CVPR2021 DDL, still rank 1st in term of Accuracy now), and based on the proposed method, we also achieve the 1st place in the leaderboard of SemanticKITTI panoptic segmentation.

Installation

Requirements

PyTorch >= 1.2
yaml
Cython
torch-scatter
nuScenes-devkit (optional for nuScenes)
spconv (tested with spconv==1.2.1 and cuda==10.2)

Data Preparation

SemanticKITTI

./
├── 
├── ...
└── path_to_data_shown_in_config/
    ├──sequences
        ├── 00/           
        │   ├── velodyne/	
        |   |	├── 000000.bin
        |   |	├── 000001.bin
        |   |	└── ...
        │   └── labels/ 
        |       ├── 000000.label
        |       ├── 000001.label
        |       └── ...
        ├── 08/ # for validation
        ├── 11/ # 11-21 for testing
        └── 21/
	    └── ...

nuScenes

./
├── 
├── ...
└── path_to_data_shown_in_config/
		├──v1.0-trainval
		├──v1.0-test
		├──samples
		├──sweeps
		├──maps

Training

modify the config/semantickitti.yaml with your custom settings. We provide a sample yaml for SemanticKITTI
train the network by running "sh train.sh"

Training for nuScenes

Please refer to NUSCENES-GUIDE

Pretrained Models

-- We provide a pretrained model for SemanticKITTI LINK1 or LINK2 (access code: xqmi)

-- For nuScenes dataset, please refer to NUSCENES-GUIDE

Semantic segmentation demo for a folder of lidar scans

python demo_folder.py --demo-folder YOUR_FOLDER --save-folder YOUR_SAVE_FOLDER

If you want to validate with your own datasets, you need to provide labels. --demo-label-folder is optional

python demo_folder.py --demo-folder YOUR_FOLDER --save-folder YOUR_SAVE_FOLDER --demo-label-folder YOUR_LABEL_FOLDER

TODO List

Release pretrained model for nuScenes.
Support multiscan semantic segmentation.
Support more models, including PolarNet, RandLA, SequeezeV3 and etc.
Integrate LiDAR Panotic Segmentation into the codebase.

Reference

If you find our work useful in your research, please consider citing our paper:

@article{zhu2020cylindrical,
  title={Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation},
  author={Zhu, Xinge and Zhou, Hui and Wang, Tai and Hong, Fangzhou and Ma, Yuexin and Li, Wei and Li, Hongsheng and Lin, Dahua},
  journal={arXiv preprint arXiv:2011.10033},
  year={2020}
}

#for LiDAR panoptic segmentation
@article{hong2020lidar,
  title={LiDAR-based Panoptic Segmentation via Dynamic Shifting Network},
  author={Hong, Fangzhou and Zhou, Hui and Zhu, Xinge and Li, Hongsheng and Liu, Ziwei},
  journal={arXiv preprint arXiv:2011.11964},
  year={2020}
}

Acknowledgments

We thanks for the opensource codebases, PolarSeg and spconv

cylinder3d's People

Contributors

Stargazers

Watchers

Forkers

collector-m tai-wang alsun-oven trendingtechnology lisilin013 tmactmac1992 rancheng zhearing changhexingchen nlpguyz zfxu eglrp sunstarchan junhaozhang98 yvanali devin-coder ginobilinie yysijie cao-dut tops666 sj-li ashleyqtds tubbz-alt coolloveboy enginbozkurt kevinladlee jhx646018057 jhlee0513 yuhaohuang950623 rnri kashu7100 cule hviktortsoi arghyachatterjee abhigoku10 robot-ai-machinelearning medical-projects shumaoorz miniflash liuxinren456852 apisxia ammaddd iprivit jediofgever overbestfitting chulongc ptzu caoyifeng001 samux87 youngsplace0913 inkyusa klotztech eldar kysheryl guragamb yuhuang-ca lijiunderstand dsml-bryan jou-98 hiyyg hcp6897 rainfalj jlqzzz dcnsw snc6si mengdejiang curiosity654 heroism502 tum-ki derkaczda ericcurl aosheng1996 sharknow sus71 kennard123661 fatcat3z ltriess zqj2333 hualuojingye akshat-khare seungchan-mok jiangwenj02 hilnkm peterzhousz stoneshuyao suyunzzz niqbal996 robo-alex daisy-geng song-jingyu duzhiqiang2019 ccodie carl12138aka zzyzzyzz topcomma xizaoqu zhangzw12319 dengyingxu ajinkyakhoche wondersimiliar

cylinder3d's Issues

weights of conv3 in UpBlock exist NAN

I download the pretrained model for SemanticKITTI and print all parameters. Then I find the weight of conv3 in UpBlock always exist NAN like this:

cylinder_3d_spconv_seg.upBlock0.conv3.weight
tensor([[[[[-0.3146,  0.0267,  0.1047,  ...,  0.1638, -0.2747,  0.1784],
           [-0.8915, -0.2567, -2.1439,  ..., -0.0997, -0.1918, -0.8922],
           [-1.6583, -1.0413, -0.8019,  ...,  0.0865, -1.4436, -1.4832],
           ...,
           [ 1.4814,  0.8771, -2.7230,  ..., -0.1184,  2.6314,  0.6151],
           [ 0.5554, -0.6339, -0.3210,  ...,  1.5973,  0.8244,  0.9369],
           [ 0.3099, -0.3487, -0.9573,  ...,  1.8343,  2.0645, -1.4957]],

          [[ 0.2434,  0.6077,  0.2401,  ...,  0.6087,  0.2166,  0.3725],
           [-0.5271, -0.3993, -1.2025,  ...,  0.6850, -0.9096, -0.5193],
           [-1.7771,  0.8933, -0.4569,  ...,  0.0112, -0.6558, -0.3272],
           ...,
           [ 0.3616, -0.4162, -2.7452,  ..., -0.1293,  0.7318, -1.2711],
           [ 0.3186,  0.4266,  0.3109,  ...,  0.1906, -0.1706,  0.4632],
           [ 0.8191,  1.3683,  0.0891,  ...,  2.4128,  1.5446,  0.6645]],

          [[ 0.3023,  0.3807,  0.2493,  ..., -1.0415,  0.1056, -0.2603],
           [ 0.9599,  0.7323,  0.1478,  ...,  0.7111, -0.1298,  0.8966],
           [ 0.7723,  0.6026, -0.1770,  ...,  0.5635,  1.8141, -0.8636],
           ...,
           [-1.4419, -1.1362, -2.6401,  ...,  0.6550,  0.7862, -0.9430],
           [-0.9298,  0.6791, -1.7838,  ...,  0.9366,  0.2681, -1.3272],
           [ 1.1701,  1.1824, -1.6827,  ...,  2.5493,  0.9161,  0.3196]]],


         [[[ 0.4152, -0.6454,  1.0213,  ..., -0.5271,  0.8759,  0.3317],
           [ 0.0152, -0.8058, -0.5352,  ..., -2.3797,  0.0426,  0.2902],
           [-2.8191, -2.3472, -0.2284,  ...,  0.1618, -0.2339, -2.1502],
           ...,
           [ 1.9137, -0.4314, -0.4321,  ...,  1.2808,  1.4155,  2.1384],
           [ 0.5110, -0.2770, -0.5622,  ...,  1.9154, -1.1110, -0.0050],
           [-0.8146,  0.8423, -1.0469,  ...,  1.8412,  0.3185, -0.1781]],

          [[ 1.3330, -0.0630,  0.4452,  ...,  0.8059,  1.5876,  0.0179],
           [ 1.3694, -0.9363, -1.4790,  ...,  0.6796,  1.1329,  1.5354],
           [-1.9959, -0.3877, -0.4930,  ...,  0.6501, -0.9343,  0.1375],
           ...,
           [ 0.7912, -1.1728,  0.4941,  ..., -0.0465, -1.7027, -0.0237],
           [-1.0025,  1.3098, -0.0768,  ..., -0.4612, -3.4584, -1.4110],
           [-0.4183,  1.0766,  0.1007,  ...,  1.8095, -0.3510, -1.2115]],

          [[ 0.4813, -1.0362,  0.1347,  ..., -0.8623, -0.3301, -1.9667],
           [ 1.4352, -1.1810,  0.5029,  ...,  0.8198,  0.6615, -0.6821],
           [ 0.1441,  0.6090,  0.3326,  ...,  0.7842, -0.8729, -1.4596],
           ...,
           [-0.1876, -0.9668, -0.5698,  ..., -0.4510,  0.0355, -0.0828],
           [-0.8910, -0.0146, -1.5001,  ...,  0.1596,  0.1089, -0.6375],
           [ 0.9165,  0.4500, -1.9313,  ...,  1.6017, -0.0742,  0.9825]]],


         [[[-0.6726, -0.7895,  0.6720,  ...,  0.1075,  0.2664,  0.1012],
           [ 0.0861,  0.4065, -0.6354,  ...,  0.1761, -0.2696, -0.0994],
           [-1.8868, -1.5101, -0.5280,  ...,  1.5993,  0.7808, -0.4933],
           ...,
           [ 2.0131, -0.2112,  0.7153,  ...,  0.8866,  1.1386,  1.9173],
           [ 0.2099, -0.9653, -0.4816,  ...,  1.1221, -1.0017,  0.2882],
           [ 0.6900,  0.0487, -1.6519,  ...,  0.9704,  0.1471, -0.7587]],

          [[ 0.0492, -0.5954, -0.1301,  ...,  0.0145,  0.4446, -0.2605],
           [ 0.8087, -0.4472, -1.0978,  ...,  0.2893,  0.1475,  0.2756],
           [-0.8992, -0.6114, -1.1732,  ...,  0.4818,  0.1650, -0.3893],
           ...,
           [ 0.7977, -0.9973,  2.3349,  ...,  0.5786, -0.1661,  0.5659],
           [ 0.2593,  1.7855,  0.0986,  ...,  0.1885, -1.8025, -0.5972],
           [-0.3040,  0.7271, -0.2476,  ...,  0.4204,  1.4570, -1.3176]],

          [[ 0.6191, -0.9954, -0.2723,  ..., -1.9767,  0.3893, -1.4162],
           [ 1.4961, -1.1198, -0.1618,  ...,  0.9444,  0.0264,  0.3396],
           [ 1.1053,  1.5829,  1.0865,  ...,  0.8140,  0.0177, -0.9481],
           ...,
           [-0.9551,  0.2058,  0.6879,  ..., -1.3027, -0.5232, -0.9984],
           [-1.3750, -0.3583, -0.7769,  ...,  0.5739, -0.3673, -1.2868],
           [ 1.1416, -0.3617, -1.5134,  ...,  2.5882,  0.1847,  0.5558]]]],



        [[[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]]],


         [[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]]],


         [[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]]]],



        [[[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]]],


         [[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]]],


         [[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]],

          [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           ...,
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
           [    nan,     nan,     nan,  ...,     nan,     nan,     nan]]]]],
       device='cuda:0')

I'm very confused about it. Could you explain why?

Question about tta

Thanks for you opening-source of this project !

But I can't search tta in google, what is tta?

Looking forward to your reply.

目前我们只提供了满足val上结果的pretrain模型，test上的结果还需要进一步的后处理，包括tta，finetuning with validation等。
Originally posted by @xinge008 in #14 (comment)

Can you have a chance to tell me the exact environment information while working on this project? Currently, I am trying to run this model in an HPC cluster, however, it seems that the combination of spconv, torch, and torch-scatter is very sensitive. Meaning that their compatible CUDA version and etc. create a lot of headaches. However, I believe I can replicate your results, if I may know your system information since I will be able to install exact dependencies.

By the way, I really like to express that it is an amazing project, I sincerely congratulate you and I wish you the continuation of this success.

Thanks in adcanve.

Model inference time?

Hi @xinge008

Thanks for your brailliant work.
One thing I'm curious is how the memory occupacy and inference time of this Cylinder3D model with 10w points input?

Best Regards!

About training parameters?

Thanks for sharing this brilliant work.

I found in the code that the batch size is 2 but training is based on the only one GPU.

I wonder what is the real batch size that could reprodece the best result in your paper.

I am looking forward to your reply.

details about point-wise refinement module

Hello,

Thanks for sharing the code.

I want to add the point-wise refinement module mentioned in the paper to the network, could you tell me some more details about this module like channels etc.? Looking forward to your reply : )

The gap between the grid representation and the point representation.

The return of this function (Asymm_3d_spconv) is still a dense representation of SparseConvTensor, but how do we restore it to a point-wise representation?

Cylinder3D/network/segmentator_3d_asymm_spconv.py

Line 249 in a1cc7a7

class Asymm_3d_spconv(nn.Module):

However, I found in this line that the return value of the model is directly used as the predicted label. Shouldn't the predicted label be point-wise? The gap between the grid representation and the point representation makes me feel strange.

Cylinder3D/train_cylinder_asym.py

Line 96 in a1cc7a7

predict_labels = my_model(val_pt_fea_ten, val_grid_ten, val_batch_size)

Could you give me some hints? Thanks.

How to train multipleScan experiments on Semantic-KITTI

Hi, thanks for sharing your great work!

I have been tested your train.sh to run single-scan experiments on Semantic-KITTI, but how should I run multi-scan experiments?

I notice that you add the multi-scan dataloader code on PC_dataset.py (

Cylinder3D/dataloader/pc_dataset.py

Line 130 in 6dfbbfa

class SemKITTI_sk_multiscan(data.Dataset):

). I tried to modify the pc_dataset_type from "SemKITTI_sk" to "SemKITTI_sk_multiscan", but the train.sh failed.

If I hope to train Cyinlder3D on multi-scan LiDAR data, how should I do?

The division of Semantic Kitti data.

Thank you for your work. I'm wondering if you process the entire 100mx100mx10m input directly during training or do you need to pre-divide it like other networks (e.g. PointASNL is first divided into 10mx10mx6m)?I think your polar based 3D division is directly applied to the whole scan, I don't know if that's right.

Cannot find point-wise prediction and loss function.

Thanks for sharing your kind code.
Unfortunately i couldn't find out the code about point-wise prediction and corresponding loss function.
I found out the output of model is [#bs, 20, 480,360,32] which is a form of voxel.
And the corresponding loss is calculated with voxel-wise class label.

Where can i find the lines about point-wise prediction and corresponding loss function?

Thanks,

Training speed

It takes Quadro GV100 5 hours for 1 epoch. Anybody trained the model on RTX3090? I failed to compile SpConv lib on 3090.

当batch_size>1时，不同batch的点一起输入到MLP？

Cylinder3D/network/cylinder_fea_generator.py
line77: processed_cat_pt_fea = self.PPmodel(cat_pt_fea)
当batch_size>1时，cat_pt_fea是不同batch的点shuffle后的集合。
不同batch的点经过shuffle后一起输入到MLP？这样是否有问题？

How to build 3D Detection target in cylindrical view?

For 3D detection, we get [x, y, z, w, l, h, vx, vy, theta]

x, y,z should be transformed into the cylindrical view. But what about the remained regression targets? Do you just regress to the original value or some transformation?

精度问题

请问这代码的精度是leaderboard 上的精度吗？

Why is your code different from the description in the paper？

Thank you very much for making this job public, but I have a few questions:
Why is your code different from the description in the paper？

Asymmetrical 3D convolution is not used in the decoder part and there are two extra 3x3x3 conv blocks.
There is no point refinement module.

Dockerfile

Could you please provide a Dockerfile which also contain spconv module?

Explain Cylindrical Projection

I'm really interested in running some test of you cylindrical projection with non cylindrical voxeling but I'm struggling finding where do you use the vexels classes in you code. If I would like to apply your cylindrical voxelization only how can I do? To be clear I know how to do cylindrical projection, I'm interested in the cylindrical partition and how it works in detail and also where is done on your code, I suppose is this part here from 230

Cylinder3D/dataloader/dataset_semantickitti.py

Line 230 in 6dfbbfa

xyz_pol = cart2polar(xyz)

but it's not clear what are you doing both for points and labels as in the paper there is no appendix regarding math operations and the repo is not well documented.

Thank you in advance

tee: logs_dir/cylinder_asym_networks_logs_tee.txt: No such file or directory ?

Thanks for releasing the codes.
With following your instruction, i ran the train.sh script.
And i found following message, but the code is still working thankfully.
Is there more setting that i miss?

Further, what does the bar mean? ( 245 / 4247 )
'4247' means number of scans per epoch?

Input Data format for spconv modules

I have a question regarding how to structure the point cloud in order to feed it into the spconv different modules. As the official repository seems to be inactive I'm asking you how to prepara a point cloud. That' my attempt:

    ptc = load_velo_scan(data[0])[:, 0:3] # Load the point cloud

    # ------------------ #
    #  Data Preparation  #
    # ------------------ #
    xyzmin = np.floor(np.percentile(ptc, 0, axis=0)).astype(np.int32)
    xyzmax = np.ceil(np.percentile(ptc, 100, axis=0)).astype(np.int32)
    ori_points = torch.tensor(ptc, requires_grad=True)
    voxel_generator = spconv.utils.VoxelGeneratorV2(voxel_size=[0.5] * 3,
                                                    point_cloud_range=[xyzmin[0], xyzmin[1], xyzmin[2], xyzmax[0], xyzmax[1], xyzmax[2]],
                                                    max_num_points=50,
                                                    max_voxels=20000,
                                                    full_mean=False)

    # ------------------------------- #
    #  Associated with compute graph  #
    # ------------------------------- #
    # 1. retain raw indice.
    ori_points_ = ori_points.detach()
    batch_point_indice = torch.arange(0, ori_points_.shape[0]).float().unsqueeze(1)  # [N, 1]
    # [N,3] -> [N,4], the add '1' is the indice in N
    ori_points_ = torch.cat((ori_points_, batch_point_indice), dim=1).view(-1, 4)
    # 2. generate voxels
    # features[num_voxels, point_per_voxel, 3+1] 1(indice), coords: [num_voxels, 1+3] 1(batch)
    # res contains:
    #   voxels: [M, max_points, ndim] float tensor. only contain points.
    #   coordinates: [M, 3] int32 tensor. zyx format.
    #   num_points_per_voxel: [M] int32 tensor.
    res = voxel_generator.generate(ori_points_.numpy())

    # https://github.com/traveller59/spconv/issues/142
    # https://github.com/traveller59/spconv/issues/188

    # features: [num_voxels, max_num_points_per_voxel, 3]
    gen_fea = torch.tensor(res['voxels'], dtype=torch.float32)
    # coors: [num_voxels, 3]
    gen_coords = torch.tensor(res['coordinates'], dtype=torch.int32)
    batch_idx = torch.zeros((gen_coords.shape[0], 1), dtype=torch.int32)  # batch=0
    gen_coords = torch.cat((batch_idx, gen_coords), dim=1)
    # # num_points: [num_voxels]
    gen_npoints = torch.tensor(res['num_points_per_voxel'], dtype=torch.int32)
    # 3. reconnect with compute graph
    gen_fea[:, :, :3] = ori_points[gen_fea[:, :, 3].long(), :]
    # # features[num_voxels, point_per_voxel, 3], N.B. set pos value to 0 in indice 0 (filling value)
    gen_fea = gen_fea[:, :, :3]
    # 4. you can do anything
    # voxel_vfe: [num_voxels, 3]
    voxel_vfe = torch.mean(gen_fea, dim=1)
    detached_vox = voxel_vfe.detach().numpy()
    spatial_shape = np.ceil(np.percentile(detached_vox, 100, axis=0) - np.percentile(detached_vox, 0, axis=0)).astype(np.int32)
    voxel_feature = spconv.SparseConvTensor(features=voxel_vfe, indices=gen_coords,
                                            spatial_shape=[400,400,400], # Don't know how to set it !!!!
                                            batch_size=1)

    simple_net = spconv.SubMConv3d(in_channels=3, out_channels=32,
                                   kernel_size=3, stride=1)
    output = simple_net(voxel_feature).dense()

If this procedure is not straightforward to you, cound you please provide an example of data preparing for a very simple network? Just to clarify it, thanks

details about point-wise refinement module

Hello,

Thanks for sharing the code.

I want to add the point-wise refinement module mentioned in the paper to the network, could you tell me some more details about this module like channels etc.? Looking forward to your reply : )

(I'm sorry I sent the same issue twice. It seems I can't delete this issue, so I close it.)

Low performance of motorcyclist and other-ground classes

Hello,

For the pre-trained model included, I noticed that performance on the classes motorcyclist and ground-other are close to zero, while in the paper their reported accuracy is much higher. Is this because of the lack of point refinement module in this repository, or is there some other explanation as to the significant difference in result here?

how the model speed?

Any speed comparsion with other sota models?

nuScenes configurations release plan?

Hi Xinge,

Thanks for releasing your code, awesome work! Do you have any plan to release the nuScenes dataset configurations?

Multi-GPU Training

Is there any way to train on multiple GPUs with your code? Cheers!

performance of SemanticKitti pre-trained model

Hi, I just ran an evaluation using the pre-trained model of SemanticKitti, the results seem to be very low, compared with the number in the paper. That's so strange. I'm using torch1.3, is it a bug about the version of Pytorch? Did anyone meet this before? @xinge008

Code Release!!

When can we expect the code to be released!!!

Question about training

What type of GPU did u use and how long did u train on the SemanticKITTI?

multiscan iou 低于 single scan

你好，为什么multiscan iou 会低于 single scan

Nuscenes Minidataset

@xinge008 Thanks for sharing the code base i have the following queries

1.To generate the .pkl files should we always use SECOND code base as you have mentioned
2.To save the predictions along with label should we follow the same steps mentioned in polar seg ?

Support will be highly appreciated !!!

Error in shape

I'm running now your code with no modifications. Just using CPU to do some tests and I get the following:

First I get this error if I set the device to CPU:

root@f125ceb56018:/working_dir# python -m cylindernet.train_cylinder_asym
/usr/local/lib/python3.8/dist-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
/working_dir/cylindernet/train_cylinder_asym.py
[480 360  32]
  0%|                                                                                                                 | 0/502 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/working_dir/cylindernet/train_cylinder_asym.py", line 167, in <module>
    main(args)
  File "/working_dir/cylindernet/train_cylinder_asym.py", line 133, in main
    outputs = my_model(train_pt_fea_ten, train_vox_ten, train_batch_size)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/working_dir/cylindernet/network/cylinder_spconv_3d.py", line 42, in forward
    coords, features_3d = self.cylinder_3d_generator(train_pt_fea_ten, train_vox_ten)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/working_dir/cylindernet/network/cylinder_fea_generator.py", line 68, in forward
    shuffled_ind = torch.randperm(pt_num, device=cur_dev)
RuntimeError: Device index must not be negative
  0%|

If I manually set cur_dev to None I get this error:

root@f125ceb56018:/working_dir# python -m cylindernet.train_cylinder_asym
/usr/local/lib/python3.8/dist-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
/working_dir/cylindernet/train_cylinder_asym.py
[480 360  32]
  0%|                                                                                                                 | 0/502 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/working_dir/cylindernet/train_cylinder_asym.py", line 167, in <module>
    main(args)
  File "/working_dir/cylindernet/train_cylinder_asym.py", line 83, in main
    for i_iter, (_, train_vox_label, train_grid, _, train_pt_fea) in enumerate(train_dataset_loader):
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/working_dir/cylindernet/dataloader/dataset_semantickitti.py", line 197, in __getitem__
    data = self.point_cloud_dataset[index]
  File "/working_dir/cylindernet/dataloader/pc_dataset.py", line 56, in __getitem__
    raw_data = np.fromfile(self.im_idx[index], dtype=np.float32).reshape((-1, 4))
ValueError: cannot reshape array of size 8193 into shape (4)

How should I evaluate the model on the test set? You don’t seem to give the code for the test.

run `train_cylinder_asym.py` error

Thanks for your great work!
When I run train.sh to infer the semantic kitti dataset, the following error is occured.

$ ./train.sh
Traceback (most recent call last):
  File "train_cylinder_asym.py", line 16, in <module>
    from dataloader.pc_dataset import get_SemKITTI_label_name
  File "/home/nrsl/Cylinder3D/dataloader/pc_dataset.py", line 18
    assert name not in REGISTERED_PC_DATASET_CLASSES, f"exist class: {REGISTERED_PC_DATASET_CLASSES}"
                                                                                                    ^
SyntaxError: invalid syntax

Point-Wise Refinement Module Missing

Hello,

After looking through the code I can't seem to find where the point-wise refinement module has been implemented; is this currently in the codebase, or is it currently absent?

Thanks,
Joshua

Trained Model output resutl error

@xinge008 Hi i ran the inference module using the shared pretrained model and I am obtaining the results as shown below

My environment is python=3.6 pytorch=1.7 cuda=10.1 spconv=1.2.1 , can you let me know the exact versions which the model had been trained on . THanks in advance

Dataparralel Issue

Hi, it's a great work!
Since the batch size can only be set to 1 when running the code on a 1080Ti GPU, I'm trying to use the data parralel trick to increase the number of training batch_size. However, I found when I set the batchsize to 2, the outputs of the model became a tensor of the shape ([4,20,480,360,32]) while the label is still ([2,480,360,32]). This conflict would rise an error of size mismatching.
Have you meet this problem before? Why would this happen?

About shared memory

Hi xinge，thanks for your amazing work and your code shared. While i train the cylinder_asym, an error about shared memory limited occurs. So i wander what size of shared memory you suggested while training? BTW, i am training on docker and only one 1080Ti available.

Help with the point refinement module

Hello,

I'm trying to implement the point refinement module following your instructions in previous issue posts, but my attempts have all resulted in stark drop in performance for the final network. Would it be possible for you to provide screencaps of how your unreleased code:
a) Gets features from the earlier linear layers (PPModel in cylinder_fea_generator) to pass to the refinement module
b) Defines the layers in the refinement module
c) Integrates the refinement module in the forward pass

If you could do this it would be a massive help! Thanks!

training time

Thanks for your amazing work, and I'm care about the time of training consuming.

My GPU is RTX 3090(single), I found the maximum of memory occupancy is around 16G, and each iteration consumes around 1.5~2s. And the iteration times for each epoch is close to 10000, repeating 40 epochs for training, it seems time-consuming.

Would you like to share your device setting and the details of time consuming during training(like iteration time and the whole training pipeline, which is mixed with the evaluation)?

Inference on CPU

@xinge008 thanks for sharing the code i have few queries
1.Can we run the inference pipeline on the CPU , what are the modifications required to run it in the cpu
2.what is the latency time in cpu and gpu
THanks in advance

Environment error

Can you please specify the environment that you used in more details ?
I am trying to build a conda env with python= 3.6, pytorch=1.7, and spconv=1.2.1 (cuda 11.0) with no success.

Namespace(config_path='config/semantickitti.yaml')
[480 360  32]
matched parameter sets: 274, and no matched: 0
Traceback (most recent call last):
  File "test_cylinder_asym.py", line 113, in <module>
    main(args)
  File "test_cylinder_asym.py", line 83, in main
    predict_labels = my_model(val_pt_fea_ten, val_grid_ten, val_batch_size)
  File "/home/*/anaconda3/envs/cylinder3d/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/*/cylinder_ws/Cylinder3D/network/cylinder_spconv_3d.py", line 44, in forward
    spatial_features = self.cylinder_3d_spconv_seg(features_3d, coords, batch_size)
  File "/home/*/anaconda3/envs/cylinder3d/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/*/cylinder_ws/Cylinder3D/network/segmentator_3d_asymm_spconv.py", line 290, in forward
    ret = self.downCntx(ret)
  File "/home/*/anaconda3/envs/cylinder3d/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/*/cylinder_ws/Cylinder3D/network/segmentator_3d_asymm_spconv.py", line 74, in forward
    shortcut = self.conv1(x)
  File "/home/*/anaconda3/envs/cylinder3d/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/*/anaconda3/envs/cylinder3d/lib/python3.6/site-packages/spconv/conv.py", line 181, in forward
    use_hash=self.use_hash)
  File "/home/*/anaconda3/envs/cylinder3d/lib/python3.6/site-packages/spconv/ops.py", line 95, in get_indice_pairs
    int(use_hash))
ValueError: /home/*/cylinder_ws/spconv/src/spconv/spconv_ops.cc 87
unknown device type

A question

Hello ,
I have read through the paper. The work looks fantastic.
I was wondering whether this network is particularly suiting for my needs or its an overkill.

I have a pointcloud map that is automatically generated and labeled from simulation. I sematic segment the cloud to two class, ground and non-ground.
The cloud doesn't have the features of LIDAR pointcloud such as sparsity in distant regions. My cloud is uniformly sampled almost equal density at any point,
will this method(Cylinder3D ) still output state-of-the-art results for such a case ?

whats your thoughts ?

Thank you very much

test.py

@xinge008 Thank you so much for sharing the code.

I am currently implementing your code repeatedly. Due to the limitation of 2080Ti, the memory during training is not enough and the batch size can only be set to 1. I want to test my training results and consider whether it needs to be implemented on 3090, but I did not find a test .py file, I don’t know if I can use PolarSeg/test_pretrain_SemanticKITTI.py to test.

About the weight of classes.

How do you determine the weight of classes in CE loss? Thx.

Baseline architecture

Hello,

I was reading your paper and quite intrigued by the new method you came up with!
However, I am quite surprised by your UNet-baseline achieving 61.5% mIoU on KITTI.
Would you mind sharing your architecture and training details for the baseline?

Best,
Daniel

Does Spconv work with CPU?

I'm building a small network following your code which is the following:

for i_iter, (_, train_vox_label, train_grid, _, train_pt_fea) in enumerate(train_dataset_loader):

    train_pt_fea_ten = [torch.from_numpy(i).type(torch.FloatTensor).to(pytorch_device) for i in return_fea]
    # train_grid_ten = [torch.from_numpy(i[:,:2]).to(pytorch_device) for i in train_grid]
    train_vox_ten = [torch.from_numpy(i).to(pytorch_device) for i in grid_ind]
    point_label_tensor = torch.Tensor(processed_label).type(torch.LongTensor).to(pytorch_device)

    ret = spconv.SparseConvTensor(train_pt_fea_ten,
                                  train_vox_ten,
                                  grid_size.shape,
                                  1)

    model = TestModel(16, 32)

where TestModel is built as follows:

class TestModel(nn.Module):
    def __init__(self, in_filters, out_filters):
        super(TestModel, self).__init__()
        self.mod = spconv.SubMConv3d(in_filters, out_filters, kernel_size=(3, 3, 3))

    def forward(self, x):
        mod_out = self.mod(x)
        return mod_out

However when I try to test this code I get a strange error from spconv module:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/working_dir/mymodel/data_processing.py", line 176, in <module>
    out = model(ret)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/working_dir/mymodel/testmodel.py", line 11, in forward
    mod_out = self.mod(x)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/spconv-1.2.1-py3.8-linux-x86_64.egg/spconv/conv.py", line 129, in forward
    device = features.device
AttributeError: 'list' object has no attribute 'device'

Anyway I think that is related with the input data format. I need to built a network with only these modules, in particolar looking at the forward module function of spconv:

 def forward(self, input):
        assert isinstance(input, spconv.SparseConvTensor)
        features = input.features
        device = features.device
        indices = input.indices
        spatial_shape = input.spatial_shape
        batch_size = input.batch_size

Expected mIoU of the pretrained checkpoint

Hello,

Would you be able to tell us what the expected mIoU of the pretrained checkpoint you supply should be? In the environment I'm using I got a mIoU of 66.914, which is significantly higher than what you report in your paper. Some clarification on why that might be the case would be very helpful.

Thanks!

Inference code to evaluate model on pointcloud files

@xinge008
Do you have Inference code to evaluate model on pointcloud files?

Can you release the code about the test submission?

Thanks for your great work. Can you release the code about the test submission on SemanticKITTI dataset? thank you

code on 2D backbone

In Section 4.2 of the Cylinder3D paper, it was mentioned that 'cylinder3d voxelization + our 2d backbone' still achieves a good performance. I wonder how 2d backbone is implemented. The paper says 3d backbone uses 3d conv while 2d backbone uses 2d conv, but what about the asymmetircal residual block and DDCM?

What defines input value?

I found that the dimension for the both inputs, train_pt_fea_ten: [126804,9], train_vox_ten: [126804,3]
I guess the 126804 represents the number of points, and i want to figure out what define both '9' and '3'.

Thanks,

xinge008 / cylinder3d Goto Github PK

cylinder3d's Introduction

Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation

News

Installation

Requirements

Data Preparation

SemanticKITTI

nuScenes

Training

Training for nuScenes

Pretrained Models

Semantic segmentation demo for a folder of lidar scans

TODO List

Reference

Acknowledgments

cylinder3d's People

Contributors

Stargazers

Watchers

Forkers

cylinder3d's Issues

Recommend Projects

Recommend Topics

Recommend Org