I trained the code on the local sever(TITAN RTX), the result is worse than the paper reported. I got some WARNING, that may make the performance decrease. How could I solve this?
2021-09-18 18:13:00,168 INFO ==> Loading parameters from checkpoint ../output/kitti_models/pvrcnn_fsa/default/ckpt/checkpoint_epoch_80.pth to GPU
2021-09-18 18:13:00,259 INFO ==> Checkpoint trained from version: pcdet+0.3.0+a7cf536
2021-09-18 18:13:00,764 INFO ==> Done (loaded 382/382)
2021-09-18 18:13:00,778 INFO *************** EPOCH 80 EVALUATION *****************
eval: 0%| | 0/943 [00:00<?, ?it/s][WARN] Cannot find rule for <class 'pcdet.models.backbones_3d.vfe.mean_vfe.MeanVFE'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'spconv.conv.SubMConv3d'>. Treat it as zero Macs and zero Params.
[INFO] Register count_bn() for <class 'torch.nn.modules.batchnorm.BatchNorm1d'>.
[INFO] Register zero_ops() for <class 'torch.nn.modules.activation.ReLU'>.
[WARN] Cannot find rule for <class 'spconv.modules.SparseSequential'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'spconv.conv.SparseConv3d'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.models.backbones_3d.spconv_backbone.SlimVoxelBackBone8x'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.models.backbones_2d.map_to_bev.height_compression.HeightCompression'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.ops.pointnet2.pointnet2_stack.pointnet2_utils.QueryAndGroup'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'torch.nn.modules.container.ModuleList'>. Treat it as zero Macs and zero Params.
[INFO] Register count_convNd() for <class 'torch.nn.modules.conv.Conv2d'>.
[INFO] Register count_bn() for <class 'torch.nn.modules.batchnorm.BatchNorm2d'>.
[WARN] Cannot find rule for <class 'torch.nn.modules.container.Sequential'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.ops.pointnet2.pointnet2_stack.pointnet2_modules.StackSAModuleMSG'>. Treat it as zero Macs and zero Params.
[INFO] Register count_linear() for <class 'torch.nn.modules.linear.Linear'>.
[INFO] Register count_convNd() for <class 'torch.nn.modules.conv.Conv1d'>.
[WARN] Cannot find rule for <class 'torch.nn.modules.normalization.GroupNorm'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'torch.nn.modules.activation.Softmax'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.models.backbones_3d.sa_block.SA_block'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.models.backbones_3d.pfe.sa_voxel_set_abstraction.SAVoxelSetAbstraction'>. Treat it as zero Macs and zero Params.
[INFO] Register zero_ops() for <class 'torch.nn.modules.padding.ZeroPad2d'>.
[WARN] Cannot find rule for <class 'pcdet.models.backbones_2d.encoder_2d.bev_encoder.BaseBEVEncoder'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.models.backbones_3d.cfe.voxel_fsa.PositionalEncoding'>. Treat it as zero Macs and zero Params.
[INFO] Register zero_ops() for <class 'torch.nn.modules.dropout.Dropout'>.
[WARN] Cannot find rule for <class 'torch.nn.modules.normalization.LayerNorm'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.models.backbones_3d.cfe.voxel_fsa.VoxelContext3D_fsa'>. Treat it as zero Macs and zero Params.
[INFO] Register count_convNd() for <class 'torch.nn.modules.conv.ConvTranspose2d'>.
[WARN] Cannot find rule for <class 'pcdet.models.backbones_2d.decoder_2d.bev_decoder.ConcatVoxelDecoder'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.utils.loss_utils.SigmoidFocalClassificationLoss'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.utils.loss_utils.WeightedSmoothL1Loss'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.utils.loss_utils.WeightedCrossEntropyLoss'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.models.dense_heads.anchor_head_single.AnchorHeadSingle'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.models.dense_heads.point_head_simple.PointHeadSimple'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.models.roi_heads.target_assigner.proposal_target_layer.ProposalTargetLayer'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.models.roi_heads.pvrcnn_head.PVRCNNHead'>. Treat it as zero Macs and zero Params.
[WARN] Cannot find rule for <class 'pcdet.models.detectors.pv_rcnn.PVRCNN'>. Treat it as zero Macs and zero Params.
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.)
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/nn/functional.py:3613: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode)
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/nn/functional.py:3658: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
"The default behavior for interpolate/upsample with float scale_factor changed "
Model PARAMS: , Model G-FLOPS: 9959985.0 64276345568.0
eval: 100%|█| 943/943 [05:37<00:00, 2.79it/s, recall_0.3=(16989, 17000) / 17558
2021-09-18 18:18:38,355 INFO *************** Performance of EPOCH 80 *****************
2021-09-18 18:18:38,355 INFO Generate label finished(sec_per_example: 0.0896 second).
2021-09-18 18:18:38,355 INFO recall_roi_0.3: 0.967593
2021-09-18 18:18:38,355 INFO recall_rcnn_0.3: 0.968220
2021-09-18 18:18:38,355 INFO recall_roi_0.5: 0.923112
2021-09-18 18:18:38,355 INFO recall_rcnn_0.5: 0.930003
2021-09-18 18:18:38,355 INFO recall_roi_0.7: 0.676558
2021-09-18 18:18:38,355 INFO recall_rcnn_0.7: 0.743365
2021-09-18 18:18:38,357 INFO Average predicted number of objects(3769 samples): 9.800
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/decorators.py:110: NumbaDeprecationWarning: Eager compilation of device functions is deprecated (this occurs when a signature is provided)
warn(NumbaDeprecationWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/decorators.py:110: NumbaDeprecationWarning: Eager compilation of device functions is deprecated (this occurs when a signature is provided)
warn(NumbaDeprecationWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/decorators.py:110: NumbaDeprecationWarning: Eager compilation of device functions is deprecated (this occurs when a signature is provided)
warn(NumbaDeprecationWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/decorators.py:110: NumbaDeprecationWarning: Eager compilation of device functions is deprecated (this occurs when a signature is provided)
warn(NumbaDeprecationWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/decorators.py:110: NumbaDeprecationWarning: Eager compilation of device functions is deprecated (this occurs when a signature is provided)
warn(NumbaDeprecationWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/decorators.py:110: NumbaDeprecationWarning: Eager compilation of device functions is deprecated (this occurs when a signature is provided)
warn(NumbaDeprecationWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/decorators.py:110: NumbaDeprecationWarning: Eager compilation of device functions is deprecated (this occurs when a signature is provided)
warn(NumbaDeprecationWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/decorators.py:110: NumbaDeprecationWarning: Eager compilation of device functions is deprecated (this occurs when a signature is provided)
warn(NumbaDeprecationWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/decorators.py:110: NumbaDeprecationWarning: Eager compilation of device functions is deprecated (this occurs when a signature is provided)
warn(NumbaDeprecationWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/decorators.py:110: NumbaDeprecationWarning: Eager compilation of device functions is deprecated (this occurs when a signature is provided)
warn(NumbaDeprecationWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (24) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (30) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (35) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (20) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (40) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (28) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (25) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (96) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (24) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (30) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (30) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (24) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (35) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (20) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (40) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (28) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (25) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/ds/anaconda3/envs/pcdet/lib/python3.7/site-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: Grid size (96) < 2 * SM count (144) will likely result in GPU under utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
2021-09-18 18:18:58,305 INFO Car [email protected], 0.70, 0.70:
bbox AP:95.6558, 89.2137, 88.9727
bev AP:90.0087, 87.4705, 86.9144
3d AP:89.0232, 78.7346, 78.2206
aos AP:95.61, 89.10, 88.80
Car [email protected], 0.70, 0.70:
bbox AP:97.8422, 93.7267, 91.9560
bev AP:92.8356, 88.4083, 88.2203
3d AP:91.2983, 82.0211, 79.9662
aos AP:97.81, 93.58, 91.75
Car [email protected], 0.50, 0.50:
bbox AP:95.6558, 89.2137, 88.9727
bev AP:95.7055, 89.2747, 89.0953
3d AP:95.6466, 89.2571, 89.0537
aos AP:95.61, 89.10, 88.80
Car [email protected], 0.50, 0.50:
bbox AP:97.8422, 93.7267, 91.9560
bev AP:97.8848, 94.1767, 94.1172
3d AP:97.8545, 94.0966, 93.9559
aos AP:97.81, 93.58, 91.75
Pedestrian [email protected], 0.50, 0.50:
bbox AP:71.9394, 66.0419, 63.8174
bev AP:67.1435, 60.5519, 56.4955
3d AP:65.0513, 57.2925, 53.5970
aos AP:64.38, 59.06, 56.54
Pedestrian [email protected], 0.50, 0.50:
bbox AP:72.2893, 67.0041, 64.0620
bev AP:67.4582, 59.9565, 55.6927
3d AP:65.0522, 56.9932, 52.4130
aos AP:63.73, 58.78, 55.69
Pedestrian [email protected], 0.25, 0.25:
bbox AP:71.9394, 66.0419, 63.8174
bev AP:76.9071, 73.0327, 70.4967
3d AP:76.8903, 73.0165, 70.3640
aos AP:64.38, 59.06, 56.54
Pedestrian [email protected], 0.25, 0.25:
bbox AP:72.2893, 67.0041, 64.0620
bev AP:78.6493, 74.0522, 71.0445
3d AP:78.6324, 74.0364, 70.9557
aos AP:63.73, 58.78, 55.69
Cyclist [email protected], 0.50, 0.50:
bbox AP:89.5657, 76.8325, 75.1482
bev AP:87.4041, 72.6511, 69.3511
3d AP:85.7404, 70.5434, 64.8741
aos AP:89.38, 76.23, 74.44
Cyclist [email protected], 0.50, 0.50:
bbox AP:94.6328, 79.5327, 75.5670
bev AP:90.2565, 73.6993, 69.4862
3d AP:88.3381, 70.5201, 66.3709
aos AP:94.37, 78.81, 74.86
Cyclist [email protected], 0.25, 0.25:
bbox AP:89.5657, 76.8325, 75.1482
bev AP:88.4720, 73.7940, 72.0709
3d AP:88.4720, 73.7940, 72.0709
aos AP:89.38, 76.23, 74.44
Cyclist [email protected], 0.25, 0.25:
bbox AP:94.6328, 79.5327, 75.5670
bev AP:93.1332, 76.1067, 73.0396
3d AP:93.1332, 76.1067, 73.0396
aos AP:94.37, 78.81, 74.86
2021-09-18 18:18:58,309 INFO Result is save to /home/ds/anaconda3/envs/pcdet/SA-Det3D/OpenPCDet/output/kitti_models/pvrcnn_fsa/default/eval/epoch_80/val/default
2021-09-18 18:18:58,309 INFO ****************Evaluation done.*****************