cvmi-lab / st3d Goto Github PK

(CVPR 2021 & T-PAMI 2022) ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection & ST3D++: Denoised Self-training for Unsupervised Domain Adaptation on 3D Object Detection

License: Apache License 2.0

Python 83.99% C++ 5.71% Cuda 9.57% C 0.48% Shell 0.24%

st3d's People

Contributors

Stargazers

Watchers

Forkers

mfkiwl collector-m canqin001 fengqiliu1221 jlqzzz deeptibhegde s-ryosky neophack trendingtechnology chinitaberrio queenie88 yeyang1021 sjtutsb tianhaofu synsin0 matthew-leng lihua213 bobrown shashankag14 leozyc-waseda jihanyang nagisa-koyama zhuoxiao-chen cv-det sky-fly97 tc5z xiaoxin83121 mengxingshifen1218 vehxianfish zhaopeisen99 darrenjkt ut-amrl xzm2002 city945 c7huang rhythmoftherain-byte yanlingz dysinia michaelzona

st3d's Issues

About the pretrain on Waymo dataset.

Hello, Jihan! First, thans for your excellent work! I am trying to reproduce the performance on Waymo->NuScenes. However I meet some questions. I wonder if you can give me some advice, thank you!

I run your codes based on OpenPCDet 0.5 and SPConv 1.2, and I made some modifications to fit OpenPCDet 0.5.
For the pretrainin stage, I use the config from ST3D/tools/cfgs/da-waymo-nus_models/secondiou/secondiou.yaml.
However, after I finish the pretrain, I test the final chekcpoint on NuScenes using the config from ST3D/tools/cfgs/da-waymo-nus_models/secondiou_st3d/secondiou_st3d.yaml, and the results are very weird:

As you can see, bbox AP and bev AP both seems reasonable, however 3d AP is almost zero, I checked my codes and have no idea why this happens. Could you please give me some insights on this problem?
I also noticed that the UPDATE_PSEUDO_LABEL_INTERVAL is 1000 in config from ST3D/tools/cfgs/da-waymo-nus_models/secondiou_st3d/secondiou_st3d.yaml, which means you only generate pseudo label once at the begining of the self-train stage. Should I modify this?

It would be very kind of you to reply me! Thank you very much!

Cannot find pseudo label for frame

I am getting an error when running train.py, it seems to have something to do with PSEUDO_LABEL not being updated. The Traceback repeats for multiple frames, not just 002080 as seen below. I've also put the full output on this gist, in case the information below is not enough. Am I missing something? Thanks for any help!

Commands Run

$ NUM_GPUS=8
$ CONFIG_FILE=cfgs/da-waymo-kitti_models/pvrcnn_st3d/pvrcnn_st3d.yaml
$ bash scripts/dist_train.sh ${NUM_GPUS} --cfg_file ${CONFIG_FILE}

Error

[2021-07-21 15:05:09,022  train.py 168  INFO]  **********************Start training da-waymo-kitti_models/pvrcnn_st3d/pvrcnn_st3d(default)**********************
generate_ps_e0: 100%|████████████████████| 232/232 [03:14<00:00,  1.19it/s, pos_ps_box=0.000(0.000), ign_ps_box=15.000(14.899)]
Traceback (most recent call last):                                                                                             
  File "train.py", line 199, in <module>
    main()
  File "train.py", line 191, in main
    ema_model=None
  File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 157, in train_model_st
    dataloader_iter=dataloader_iter, ema_model=ema_model
  File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 42, in train_one_epoch_st
    target_batch = next(dataloader_iter)
  File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
    return self._process_next_batch(batch)
  File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
  File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 413, in __getitem__
    self.fill_pseudo_labels(input_dict)
  File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/dataset.py", line 146, in fill_pseudo_labels
    gt_boxes = self_training_utils.load_ps_label(input_dict['frame_id'])
  File "/home/user5/open-mmlab/ST3D/tools/../pcdet/utils/self_training_utils.py", line 221, in load_ps_label
    raise ValueError('Cannot find pseudo label for frame: %s' % frame_id)
ValueError: Cannot find pseudo label for frame: 002080

epochs:   0%|                                                                                           | 0/30 [04:05<?, ?it/s]

Environment

Python 3.7
CUDA 10.0
PyTorch 1.1
spconv 1.0 (commit 8da6f96)
pcdet 0.2.0+73dda8c

Unable to reproduce your result

I have used the following config to pre-train a model.
https://github.com/CVMI-Lab/ST3D/blob/73dda8c131b960f6970ffd49d44d0f15f6f60f00/tools/cfgs/da-waymo-kitti_models/secondiou/secondiou_old_anchor_sn.yaml

The best evaluation results of the pre-trained model on WOD (23th epoch) were as follows:
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.4767
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.4701
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.4155
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.4096

Then, I have used the following config to self-train the model.
https://github.com/CVMI-Lab/ST3D/blob/73dda8c131b960f6970ffd49d44d0f15f6f60f00/tools/cfgs/da-waymo-kitti_models/secondiou_st3d/secondiou_st3d.yaml

The best evaluation results of the self-trained model on KITTI (11th epoch) were as follows:
Car [email protected], 0.70, 0.70:
bbox AP:88.8405, 84.5192, 84.6117
bev AP:87.7364, 78.1648, 77.4027
3d AP:68.8041, 60.3439, 59.5532
aos AP:88.73, 84.20, 84.05
Car [email protected], 0.70, 0.70:
bbox AP:93.6732, 85.0615, 84.7418
bev AP:90.1386, 81.8106, 81.0929
3d AP:68.9595, 59.3669, 57.4259
aos AP:93.54, 84.74, 84.18
Car [email protected], 0.50, 0.50:
bbox AP:88.8405, 84.5192, 84.6117
bev AP:95.3801, 88.1350, 87.7074
3d AP:95.2582, 87.8998, 87.4439
aos AP:88.73, 84.20, 84.05
Car [email protected], 0.50, 0.50:
bbox AP:93.6732, 85.0615, 84.7418
bev AP:97.4356, 92.7666, 90.8911
3d AP:97.2213, 90.8523, 90.5397
aos AP:93.54, 84.74, 84.18

some question about the paper's result

Hi, thanks for sharing so great work. For the word " We also observe that it is hard to adapt detectors from the point clouds with more LiDAR beams (e.g. Waymo) to the point clouds with fewer LiDAR beams (e.g. NuScenes), while the opposite adaptation is relatively easy" in paper section 4.2, I just think the large improvement on kitti dataset is due to kitti dataset is too small, not about more lidar beams to fewer lidar beams is hard and opposite adaptation is easy, just recomand that you can give an experiment that nuscene -> waymo' result which is more convincing。

Several problems with setup.py

Hello,
Thank you very much for this amazing work !

I have been trying to make the code (python setup.py develop) work on my windows 10, then 11, and now ubuntu, but have encountered countless errors with every environments (I've tried many Python versions, many torch, many cuda toolkits..)

Could you please detail one env list (versions for each dependencies) that is sure to function for the master branch ?
Would you also by any chance have a dockerfile to make it work ?

Up to now my trials were :

Create fresh environment (conda)
Install python, torch, torchvision, cudatoolkit and/or run "pip install -r requirements.txt"
install spconv by cloning it from OpenPCDet and running "python setup.py bdist_wheel
run "python setup.py develop"

This has always given many errors and I have never been able to have it work.
I have been able to run the code ("python setup.py develop" gave no error) on docker but not on any real machine.

Hope you have some time to answer me !

ST3D on different classes

I did the experiments on Waymo -> MyOwnDataset of ['Vehicle', 'Pedestrian', 'Cyclist'] on PVRCNN. But the result shows, after self-train, the AP of the model for 3 classes is worse than that of the model just train for single. How can I get a stable model for all 3 classes?

PVRCNN ST3D (w/ sn) model for Waymo -> KITTI task

Dear Jihan,

It seems that the model you linked to PVRCNN ST3D (w/ sn) for Waymo -> KITTI task is actually the model of Second-IoU. When possible, may you please fix that?

Many thanks!! :)

pvrcnn performance on nuscenes dataset

I have several questions about the ST3D model during my practice，thanks！

I change the config file to support multi-categories detection , adapt from Nuscenes to kitti dataset, PVRCNN trained on nuscenes cannot detection cyclist well? the mAP after 50 epoch is below:
car mean AP: 0.717315570146554
pedestrian mean AP: 0.6353227795955906
motorcycle mean AP: 0.0
truck mean AP: 0.5661849787683544
bus mean AP: 0.8888888888888891
bicycle mean AP: 0.0
construction_vehicle mean AP: 0.0
trailer mean AP: 0.0
I see that many nuscenes SOTA models such as center-point use multiframes ( max_sweep=10 ) to fusion the lidar, but the provided pvrcnn_old_anchor_ros.yaml set max_sweep=1 ？
The meaning of 'anchor_bottom_heights' of different categories? in waymmo dataset this param is set to [0], but for nuscenes different categories have much differences
Have you tested the MEMORY_ENSEMBLE in pesudo-label self-training process, i found the default setting is Enable: false.
Could you share your work at ST3D++ that combine the source domain dataset to co-training the model?
thank you!

Some question about DSBN

I noticed that, in your ST3D++ paper, the transformation parameters γ and β are shared between two domains since "the transformation parameters γ and β are domain agnostic". but in paper "Domain-specific batch normalization for unsupervised domain adaptation", γ and β are not shared between source domain and target domain. I want to confirm whether the γ and β are shared to reproduce you work. Look forward to you reply, thank you very much!

Question about ST3D applied on other dataset

Hi, thanks for you great work.

Is there any plan to apply ST3D on other dataset, such as ONCE? :)
This project alse uses ST3D, but does not achive good adaption result on nuscenes_to_once, maybe due to improper configuration.

Beside, I apply ST3D on adaption from DeepRoute dataset to ONCE dataset, the adaption AP3D result decrease to 40 from source result 55.

Can you give some advice to improve result on ONCE dataset?

Look forward to your soon reply!

How can I use it in my dataset?

@jihanyang
Thanks for your awesome work.
How can I use it in my dataset.
Do you have any suggestions? Thanks a lot. :)

Self-training on multiple GPUs with multiple workers does not work

Hello,

thank you for your nice work. I adapted your code to v0.5 of OpenPCDet and everything seems to run (only a few result checks done --> sanity checks), except in the scenario when I want to train on multiple GPUs with more than 0 workers. However, single GPU - multiple workers and Multi-GPU - zero workers are no issue.
The problem is, that when multiple workers get initialised on more than one GPU, the line

ST3D/pcdet/utils/self_training_utils.py

Line 233 in 68a9d3e

raise ValueError('Cannot find pseudo label for frame: %s' % frame_id)

causes an error. The pseudo label dicts are empty on the worker threads, but not on the main processes of each GPU. This is interesting because the single GPU training with multiple workers runs as expected.

Maybe you already know what could be the problem here.

Thank you in advance!

nuScenes-KITTI PVRCNN ST3D ckpt

Hi Jihan,

I downloaded the model ckpt for nuScenes to kitti for pvrcnn st3d and used the same config as provided in this repo. The results I get however are very high (unlike what was reported in the table on the README page of 71.11 R40 AP) which I've pasted below:

Car [email protected], 0.70, 0.70:
bbox AP:96.6183, 88.7703, 88.4658
bev  AP:88.6752, 87.4535, 87.0203
3d   AP:85.3303, 84.1034, 77.8363
aos  AP:96.47, 88.40, 88.01
Car [email protected], 0.70, 0.70:
bbox AP:97.4681, 92.1491, 91.9104
bev  AP:91.8510, 88.8938, 88.6404
3d   AP:86.5975, 83.8631, 81.8083
aos  AP:97.33, 91.73, 91.40
Car [email protected], 0.50, 0.50:
bbox AP:96.6183, 88.7703, 88.4658
bev  AP:96.8267, 88.9212, 88.6421
3d   AP:96.8180, 88.9199, 88.6381
aos  AP:96.47, 88.40, 88.01
Car [email protected], 0.50, 0.50:
bbox AP:97.4681, 92.1491, 91.9104
bev  AP:97.7262, 94.2243, 92.1346
3d   AP:97.7201, 94.2192, 92.1262
aos  AP:97.33, 91.73, 91.40

Could you help me double-check that the uploaded ckpt.pth file for the pvrcnn st3d (no SN) on nuscenes to kitti is correct? Seems like it may be an oracle checkpoint.

Segmentation fault when using "nuscenes" eval metric.

When I change the eval metric from "kitti" to "nuscenes" in the config file, after displaying the recall values, it goes to a segmentation fault. I have enabled --save_to_file. Below is the error message:

`[2021-07-21 12:34:30,972 eval_utils.py 52 INFO] *************** EPOCH 18 EVALUATION *****************
eval: 100%|██████████████████████████████| 7033/7033 [56:28<00:00, 2.08it/s, recall_0.3=(119816, 119816) / 202469]
[2021-07-21 13:30:59,219 eval_utils.py 91 INFO] *************** Performance of EPOCH 18 *****************
[2021-07-21 13:30:59,273 eval_utils.py 93 INFO] Generate label finished(sec_per_example: 0.1205 second).
[2021-07-21 13:30:59,273 eval_utils.py 109 INFO] recall_roi_0.3: 0.591775
[2021-07-21 13:30:59,273 eval_utils.py 110 INFO] recall_rcnn_0.3: 0.591775
[2021-07-21 13:30:59,273 eval_utils.py 109 INFO] recall_roi_0.5: 0.450168
[2021-07-21 13:30:59,273 eval_utils.py 110 INFO] recall_rcnn_0.5: 0.450168
[2021-07-21 13:30:59,273 eval_utils.py 109 INFO] recall_roi_0.7: 0.202480
[2021-07-21 13:30:59,274 eval_utils.py 110 INFO] recall_rcnn_0.7: 0.202480
[2021-07-21 13:30:59,294 eval_utils.py 118 INFO] Average predicted number of objects(28130 samples): 22.359
/home/labuser/anaconda3/envs/st3d/lib/python3.6/site-packages/numba/cuda/envvars.py:17: NumbaWarning:
Environment variables with the 'NUMBAPRO' prefix are deprecated and consequently ignored, found use of NUMBAPRO_CUDA_DRIVER=/usr/lib/x86_64-linux-gnu/libcuda.so.

For more information about alternatives visit: ('https://numba.pydata.org/numba-doc/latest/cuda/overview.html', '#cudatoolkit-lookup')
warnings.warn(errors.NumbaWarning(msg))
/home/labuser/anaconda3/envs/st3d/lib/python3.6/site-packages/numba/cuda/envvars.py:17: NumbaWarning:
Environment variables with the 'NUMBAPRO' prefix are deprecated and consequently ignored, found use of NUMBAPRO_NVVM=/usr/local/cuda-10.2/nvvm/lib64/libnvvm.so.

some question about table in paper

Hi, Jihan, thanks for open source the paper, I just have some question about tables in paper.
why do experiment in Table S11,S12,S13? Those experiments are evaluated use IOU=0.5,but before in Table 2,3,4 , you already do those experiment but with IOU=0.7? What's more, In Table S11, why it has opposite result with Table 2, in Table S11, source only is better that ROS, but in Table 2, ros is better than source only.

Thanks for reply.

Number of epochs for nuscenes-kitti

Hi I was just wondering how many epochs did you train on the nuscenes - kitti baseline (without SN or ROS) to get the result of 17.92 for 3D AP as reported in the paper? I've trained the secondiou_old_anchor.yaml cfg file for 3 epochs (as with ROS and SN) and only got the following results:

Car [email protected], 0.70, 0.70:
bbox AP:60.1853, 48.0934, 48.1308
bev AP:31.2696, 27.1278, 26.6632
3d AP:4.5068, 3.2667, 3.3441
aos AP:42.78, 34.73, 35.04

Infos were generated with the original OpenPCDet repo - do I need to regenerate them for KITTI and nuscenes?

Also I noticed when evaluating the nuscenes-kitti for SN and ROS, they were only trained to 3 epochs. I re-trained the nuscenes dataset with the exact same secondiou_old_anchor_ros.yaml and didn't manage to reproduce the result from the model zoo. Update: I've re-generated the infos with st3d repo and it gives a lower performance for secondiou_old_anchor_ros.yaml.

Training was done with the command

python train.py --cfg_file cfgs/da-nuscenes-kitti_models/secondiou/secondiou_old_anchor_ros.yaml --batch_size 4 --epochs 3 --extra_tag st3d_infos

Cfg file for secondiou_old_anchor_ros.yaml is as below. Dataset_config files were unchanged.

CLASS_NAMES: ['car']

DATA_CONFIG:
    _BASE_CONFIG_: cfgs/dataset_configs/da_nuscenes_kitti_dataset.yaml
    MAX_SWEEPS: 1
    PRED_VELOCITY: False
    BALANCED_RESAMPLING: False
    SHIFT_COOR: [0.0, 0.0, 1.8]
    
    DATA_AUGMENTOR:
        DISABLE_AUG_LIST: ['normalize_object_size']
        AUG_CONFIG_LIST:
            - NAME: random_object_scaling
              SCALE_UNIFORM_NOISE: [0.75, 1.0]

            - NAME: normalize_object_size
              SIZE_RES: [-0.75, -0.34, -0.2]

            - NAME: random_world_flip
              ALONG_AXIS_LIST: ['x', 'y']

            - NAME: random_world_rotation
              WORLD_ROT_ANGLE: [-0.3925, 0.3925]

            - NAME: random_world_scaling
              WORLD_SCALE_RANGE: [0.95, 1.05]

DATA_CONFIG_TAR:
    _BASE_CONFIG_: cfgs/dataset_configs/da_kitti_dataset.yaml
    TARGET: True
    FOV_POINTS_ONLY: False
    CLASS_NAMES: ['Car']
    SHIFT_COOR: [0.0, 0.0, 1.6]

MODEL:
    NAME: SECONDNetIoU

    VFE:
        NAME: MeanVFE

    BACKBONE_3D:
        NAME: VoxelBackBone8x

    MAP_TO_BEV:
        NAME: HeightCompression
        NUM_BEV_FEATURES: 256

    BACKBONE_2D:
        NAME: BaseBEVBackbone

        LAYER_NUMS: [5, 5]
        LAYER_STRIDES: [1, 2]
        NUM_FILTERS: [128, 256]
        UPSAMPLE_STRIDES: [1, 2]
        NUM_UPSAMPLE_FILTERS: [256, 256]

    DENSE_HEAD:
        NAME: AnchorHeadSingle
        CLASS_AGNOSTIC: False

        USE_DIRECTION_CLASSIFIER: True
        DIR_OFFSET: 0.78539
        DIR_LIMIT_OFFSET: 0.0
        NUM_DIR_BINS: 2

        ANCHOR_GENERATOR_CONFIG: [
            {
                'class_name': 'car',
                'anchor_sizes': [[4.2, 2.0, 1.6]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [0],
                'align_center': False,
                'feature_map_stride': 8,
                'matched_threshold': 0.55,
                'unmatched_threshold': 0.4
            }
        ]

        TARGET_ASSIGNER_CONFIG:
            NAME: AxisAlignedTargetAssigner
            POS_FRACTION: -1.0
            SAMPLE_SIZE: 512
            NORM_BY_NUM_EXAMPLES: False
            MATCH_HEIGHT: False
            BOX_CODER: ResidualCoder

        LOSS_CONFIG:
            LOSS_WEIGHTS: {
                'cls_weight': 1.0,
                'loc_weight': 2.0,
                'dir_weight': 0.2,
                'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
            }

    ROI_HEAD:
        NAME: SECONDHead
        CLASS_AGNOSTIC: True

        SHARED_FC: [256, 256]
        IOU_FC: [256, 256]
        DP_RATIO: 0.3

        NMS_CONFIG:
            TRAIN:
                NMS_TYPE: nms_gpu
                MULTI_CLASSES_NMS: False
                NMS_PRE_MAXSIZE: 9000
                NMS_POST_MAXSIZE: 512
                NMS_THRESH: 0.8
            TEST:
                NMS_TYPE: nms_gpu
                MULTI_CLASSES_NMS: False
                NMS_PRE_MAXSIZE: 1024
                NMS_POST_MAXSIZE: 100
                NMS_THRESH: 0.7

        ROI_GRID_POOL:
            GRID_SIZE: 7
            IN_CHANNEL: 512
            DOWNSAMPLE_RATIO: 8

        TARGET_CONFIG:
            BOX_CODER: ResidualCoder
            ROI_PER_IMAGE: 128
            FG_RATIO: 0.5

            SAMPLE_ROI_BY_EACH_CLASS: True
            CLS_SCORE_TYPE: raw_roi_iou

            CLS_FG_THRESH: 0.75
            CLS_BG_THRESH: 0.25
            CLS_BG_THRESH_LO: 0.1
            HARD_BG_RATIO: 0.8

            REG_FG_THRESH: 0.55

        LOSS_CONFIG:
            IOU_LOSS: BinaryCrossEntropy
            LOSS_WEIGHTS: {
                'rcnn_iou_weight': 1.0,
                'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
            }

    POST_PROCESSING:
        RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
        SCORE_THRESH: 0.1
        OUTPUT_RAW_SCORE: False

        EVAL_METRIC: kitti

        NMS_CONFIG:
            MULTI_CLASSES_NMS: False
            NMS_TYPE: nms_gpu
            NMS_THRESH: 0.01
            NMS_PRE_MAXSIZE: 4096
            NMS_POST_MAXSIZE: 500


OPTIMIZATION:
    OPTIMIZER: adam_onecycle
    LR: 0.003
    WEIGHT_DECAY: 0.01
    MOMENTUM: 0.9

    MOMS: [0.95, 0.85]
    PCT_START: 0.4
    DIV_FACTOR: 10
    DECAY_STEP_LIST: [35, 45]
    LR_DECAY: 0.1
    LR_CLIP: 0.0000001

    LR_WARMUP: False
    WARMUP_EPOCH: 1

    GRAD_NORM_CLIP: 10
``

Question about the implementation of pseudo label update

Thank you for releasing your source code.
I have a small question about the implementation of pseudo label update in QTMB.

For what purpose are the pred_cls_scores and the pred_iou_scores stored in gt_infos?
It looks like they are not used for anything.

ST3D/pcdet/utils/self_training_utils.py

Lines 189 to 194 in 73dda8c

    
           gt_infos = { 
        
               'gt_boxes': gt_box, 
        
               'cls_scores': pred_cls_scores, 
        
               'iou_scores': pred_iou_scores, 
        
               'memory_counter': np.zeros(gt_box.shape[0]) 
        
           }

How many point clouds are used when waymo is served as source domain?

Hi @jihanyang , many thanks for your excellent work.

According to the config below, 50% of the training frames are used during pre-training. Does this mean that at least about 70,000 frames of waymo point clouds are needed to fully reproduce results of waymo-to-kitti in ST3D paper? Have you tried using 100% or only 20% of the frames?

ST3D/tools/cfgs/dataset_configs/da_waymo_dataset.yaml

Lines 13 to 16 in 68a9d3e

    
           SAMPLED_INTERVAL: { 
        
               'train': 2, 
        
               'test': 5 
        
           }

BOX_FILTER usage vs POINT_CLOUD_RANGE

Hi,

Thanks for your work and ongoing responses to issues. I appreciate the support that you provide for this repo.

In your da_kitti_dataset.yaml cfg file, you specified the following:

 
DATASET: 'KittiDataset'
DATA_PATH: '../data/kitti'

POINT_CLOUD_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4]

...


DATA_PROCESSOR:
    - NAME: mask_points_and_boxes_outside_range
      REMOVE_OUTSIDE_BOXES: True

    - NAME: shuffle_points
      SHUFFLE_ENABLED: {
        'train': True,
        'test': False
      }

    - NAME: transform_points_to_voxels
      VOXEL_SIZE: [0.1, 0.1, 0.15]
      MAX_POINTS_PER_VOXEL: 5
      MAX_NUMBER_OF_VOXELS: {
        'train': 80000,
        'test': 90000
      }

TEST:
    BOX_FILTER: {
        'USE_IMAGE_AREA_FILTER': True,
        'FOV_FILTER': True,
        'LIMIT_RANGE': [-75.2, -75.2, -2, 75.2, 75.2, 4]
    }

What's the difference between the POINT_CLOUD_RANGE at the top of the cfg file, and the BOX_FILTER at the bottom? Both the nuscenes and waymo da configs don't have this TEST-BOX_FILTER parameter. It seems to me like the mask_points_and_boxes_outside_range function in DATA_PROCESSOR will filter out all the boxes and points that are outside the POINT_CLOUD_RANGE indicated above.

Looking at this line for the TEST.BOX_FILTER, it seems like the box is filtering based on FOV. However, aren't we already filtering based on FOV when we specify FOV_POINTS_ONLY in the model cfgs? For example here.

What's the usage of the BOX_FILTER and its various parameters? And why is it only for the KITTI dataset?

RCNN regression loss not calculated

Hello,
It is apparent from the code and config files that the rcnn regression loss is not calculated, although the function exists in roi_head_template.py . If this is correct, what is the motivation behind this?

Pseudo label problems on self-training of Waymo -> KITTI task

About random_object_scaling and normalize_object_size

When you use nuscenes as the target domain, I see you do not use random_object_scaling and normalize_object_size techniques. But for KITTI you use them and can you give an explanation? Thank you.

cannot reproduce your results of nuscene->kitti in v0.3.0

hi, sorry for bother you, i cannot reproduce your results of nuscene->kitti in v0.3.0. i have noticed some one reproduce your results in v0.2.0, but i do not find the git branch to run with OpenPCDet v0.2.0. could you please help me to reproduce the performance in your paper.

A question about pre-training model

Hello, I have a question for you. For the pre-training model, whether it is possible to add the target data in the pre-train stage, and use the method,such as adversarial training to train the model, so that it has better performance on the target data and can get better pseudo labels. Finally, the model can have better domain adaptability and stability. Do you have any consideration or experiment in this aspect.

ST3D学习交流微信群

SHIFT_COOR？

Hi，jihan, the parameter SHIFT_COOR in cfg file pvrcnn_old_anchor_ros.yaml means what? does SHIFT_COOR is the lidar position difference among different dataset?

Config files for multi-class self-training

Hi,

I wonder if the multi-class config (.yaml) files will be available.

In paper ST3D++: "two thresholds T_pos and T_neg are typically set as 0.6 and 0.25", but it seems that these two thresholds are set for car class only. I tried 0.6 and 0.25 for ped and cyc but the results were not good.

Could you provide the threshold setting for the other classes? It would be even better if you provide the whole config files for multi-class self-training.

Thanks.

pretrain_model performance bad

hello，jihan. Thanks for your work, when I use the pretrained model to eval the KITTI results, I got a bad results as fellows:
[2021-07-30 10:39:38,642 eval_utils.py 91 INFO] *************** Performance of EPOCH 80 *****************
[2021-07-30 10:39:38,643 eval_utils.py 93 INFO] Generate label finished(sec_per_example: 0.0255 second).
[2021-07-30 10:39:38,643 eval_utils.py 109 INFO] recall_roi_0.3: 0.977685
[2021-07-30 10:39:38,643 eval_utils.py 110 INFO] recall_rcnn_0.3: 0.977685
[2021-07-30 10:39:38,643 eval_utils.py 109 INFO] recall_roi_0.5: 0.798957
[2021-07-30 10:39:38,643 eval_utils.py 110 INFO] recall_rcnn_0.5: 0.798957
[2021-07-30 10:39:38,643 eval_utils.py 109 INFO] recall_roi_0.7: 0.070421
[2021-07-30 10:39:38,643 eval_utils.py 110 INFO] recall_rcnn_0.7: 0.070421
[2021-07-30 10:39:38,647 eval_utils.py 118 INFO] Average predicted number of objects(3769 samples): 11.060
[2021-07-30 10:39:57,848 eval_utils.py 129 INFO] Car [email protected], 0.70, 0.70:
bbox AP:17.6470, 23.7820, 33.4059
bev AP:11.3823, 14.4799, 21.0351
3d AP:0.6494, 0.5051, 9.0909
aos AP:17.53, 23.37, 32.83
Car [email protected], 0.70, 0.70:
bbox AP:18.4800, 23.7817, 28.3788
bev AP:11.2770, 13.3130, 14.3258
3d AP:0.0528, 0.1354, 0.1768
aos AP:18.36, 23.37, 27.70
Car [email protected], 0.50, 0.50:
bbox AP:17.6470, 23.7820, 33.4059
bev AP:45.1927, 54.1058, 63.2144
3d AP:32.6627, 41.2533, 48.3045
aos AP:17.53, 23.37, 32.83
Car [email protected], 0.50, 0.50:
bbox AP:18.4800, 23.7817, 28.3788
bev AP:47.9240, 55.8623, 62.0041
3d AP:33.6433, 41.2521, 45.2606
aos AP:18.36, 23.37, 27.70

And I just use the pretrianed second_iou model with kitti_models/secondiou_orcale config.

the difference between scale_ pre_ Object and normalize_ object_size

In the file augmentor_utils.py, I want to know the difference between scale_ pre_ Object and normalize_ object_size. In my opinion, they perform the same function。

def scale_pre_object(gt_boxes, points, gt_boxes_mask, scale_perturb, num_try=50):
"""
uniform sacle object with given range
Args:
gt_boxes: (N, 7) under unified coordinates
points: (M, 3 + C) points in lidar
gt_boxes_mask: (N), boolen mask for
scale_perturb:
num_try:
Returns:
"""

def normalize_object_size(boxes, points, boxes_mask, size_res):
"""
:param boxes: (N, 7) under unified boxes
:param points: (N, 3 + C)
:param boxes_mask
:param size_res: (3) [l, w, h]
:return:
"""

About SCALE_UNIFORM_NOISE and SIZE_RES

Thanks for your great work. How do your determine the hyperparameters SCALE_UNIFORM_NOISE and SIZE_RES? I find they are different in da_nuscenes_dataset.yaml and da_nuscenes_kitti_dataset.yaml. If I want to run a new setting, e.g. KITTI->nuscenes, How should I set these hyperparameters?

About ST from waymo to nus

hi, Jihan
I have try a lot of times using your lastest config files, but fail to reproduce the performance of ST from waymo to nus.
for secondiou_st, the best performance showed in No. 4 epoch.
[2021-12-02 09:53:08,270 eval_utils.py 50 INFO] *************** EPOCH 4 EVALUATION *****************
[2021-12-02 09:54:26,908 eval_utils.py 89 INFO] *************** Performance of EPOCH 4 *****************
[2021-12-02 09:54:26,908 eval_utils.py 91 INFO] Generate label finished(sec_per_example: 0.0131 second).
[2021-12-02 09:54:26,908 eval_utils.py 107 INFO] recall_roi_0.3: 0.582705
[2021-12-02 09:54:26,908 eval_utils.py 108 INFO] recall_rcnn_0.3: 0.582705
[2021-12-02 09:54:26,909 eval_utils.py 107 INFO] recall_roi_0.5: 0.476606
[2021-12-02 09:54:26,909 eval_utils.py 108 INFO] recall_rcnn_0.5: 0.476606
[2021-12-02 09:54:26,909 eval_utils.py 107 INFO] recall_roi_0.7: 0.234905
[2021-12-02 09:54:26,909 eval_utils.py 108 INFO] recall_rcnn_0.7: 0.234905
[2021-12-02 09:54:26,911 eval_utils.py 116 INFO] Average predicted number of objects(6019 samples): 5.036
[2021-12-02 09:54:37,770 eval_utils.py 127 INFO] Car [email protected], 0.70, 0.70:
bbox AP:45.3321, 45.3321, 45.3321
bev AP:30.9359, 30.9359, 30.9359
3d AP:17.2579, 17.2579, 17.2579
aos AP:26.60, 26.60, 26.60
Car [email protected], 0.70, 0.70:
bbox AP:47.2941, 47.2941, 47.2941
bev AP:26.0944, 26.0944, 26.0944
3d AP:12.9529, 12.9529, 12.9529
aos AP:27.46, 27.46, 27.46
Car [email protected], 0.50, 0.50:
bbox AP:45.3321, 45.3321, 45.3321
bev AP:34.6136, 34.6136, 34.6136
3d AP:33.8733, 33.8733, 33.8733
aos AP:26.60, 26.60, 26.60
Car [email protected], 0.50, 0.50:
bbox AP:47.2941, 47.2941, 47.2941
bev AP:34.4677, 34.4677, 34.4677
3d AP:29.9190, 29.9190, 29.9190
aos AP:27.46, 27.46, 27.46
I used 4 RTX2080TI cards, with batch_size 16.
Could you help me to find what is wrong with it?

Second-IoU or PV-RCNN source only pre-trained models

Hi,

Sorry for my trivial question. It seems that I could not find pre-trained .pth models for Second-IoU or PV-RCNN for nuScenes dataset. I'm attempting to reproduce the task nuScenes -> KITTI in Table 1 in ST3D paper, where I wish to see the result of "source only" for Second-IoU or PV-RCNN, both in numerics and visualised graphs. But I can't find the pre-trained .pth files for these 2 methods on nuScenes.

In this case, do I have to retrain them? It takes ages to retrain on my computer. If you happen to have those .pth files, would you mind uploading them? Or, please correct me if they have been uploaded somewhere.

Thanks so much!

maximum recursion depth exceeded in comparison

Hey,

I'm running the training code for the default setting of nuscenes to kitti. However, I encountered this error after 3 epochs. Can you help me with it? Did anyone face this before?

Thanks in advance,

generate_ps_e4: 100%|█| 3712/3712 [22:44<00:00, 2.72it/s, pos_ps_box=0.000(0.00
train: 0%| | 0/3712 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_trace_dispatch_regular.py", line 422, in call
if not is_stepping and frame_cache_key in cache_skips:
RecursionError: maximum recursion depth exceeded in comparison
Traceback (most recent call last):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_trace_dispatch_regular.py", line 422, in call
if not is_stepping and frame_cache_key in cache_skips:
RecursionError: maximum recursion depth exceeded in comparison
Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized
Thread 0x00007f85e9ffb700 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/pydevd.py", line 172 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85e97fa700 (most recent call first):
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 306 in wait
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 558 in wait
File "/home/fare01/.pycharm_helpers/pydev/pydevd.py", line 150 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85cffff700 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 292 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85cf7fe700 (most recent call first):
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 306 in wait
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/queue.py", line 179 in get
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 367 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Current thread 0x00007f88cc4640c0 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_trace_dispatch_regular.py", line 422 in call
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/site-packages/numpy/core/numeric.py", line 424 in count_nonzero
File "<array_function internals>", line 5 in count_nonzero
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/augmentor_utils.py", line 24 in random_flip_along_x
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/data_augmentor.py", line 93 in random_world_flip
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/data_augmentor.py", line 147 in forward
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 225 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
...
Traceback (most recent call last):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_trace_dispatch_regular.py", line 422, in call
if not is_stepping and frame_cache_key in cache_skips:
RecursionError: maximum recursion depth exceeded in comparison
Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized
Thread 0x00007f85e9ffb700 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/pydevd.py", line 172 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85e97fa700 (most recent call first):
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 306 in wait
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 558 in wait
File "/home/fare01/.pycharm_helpers/pydev/pydevd.py", line 150 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85cffff700 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 292 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85cf7fe700 (most recent call first):
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 306 in wait
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/queue.py", line 179 in get
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 367 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Current thread 0x00007f88cc4640c0 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_trace_dispatch_regular.py", line 422 in call
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/site-packages/numpy/core/numeric.py", line 424 in count_nonzero
File "<array_function internals>", line 5 in count_nonzero
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/augmentor_utils.py", line 24 in random_flip_along_x
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/data_augmentor.py", line 93 in random_world_flip
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/data_augmentor.py", line 147 in forward
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 225 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
...
epochs: 13%|▏| 4/30 [3:25:05<22:13:08, 3076.49s/it, st_loss=0.232(1.103), pos_p
Traceback (most recent call last):
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/queue.py", line 179, in get
self.not_empty.wait(remaining)
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 306, in wait
gotit = waiter.acquire(True, timeout)
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 2533) is killed by signal: Aborted.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/mnt/data/fare01/ST3D/tools/train_utils/train_st_utils.py", line 150, in train_model_st
accumulated_iter = train_one_epoch_st(
File "/mnt/data/fare01/ST3D/tools/train_utils/train_st_utils.py", line 42, in train_one_epoch_st
target_batch = next(dataloader_iter)
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
idx, data = self._get_data()
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1142, in _get_data
success, data = self._try_get_data()
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 2533) exited unexpectedly
python-BaseException
Traceback (most recent call last):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_trace_dispatch_regular.py", line 422, in call
if not is_stepping and frame_cache_key in cache_skips:
RecursionError: maximum recursion depth exceeded in comparison
Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized
Thread 0x00007f85e9ffb700 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/pydevd.py", line 172 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85e97fa700 (most recent call first):
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 306 in wait
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 558 in wait
File "/home/fare01/.pycharm_helpers/pydev/pydevd.py", line 150 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85cffff700 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 292 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85cf7fe700 (most recent call first):
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 306 in wait
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/queue.py", line 179 in get
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 367 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Current thread 0x00007f88cc4640c0 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_trace_dispatch_regular.py", line 422 in call
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/site-packages/numpy/core/numeric.py", line 424 in count_nonzero
File "<array_function internals>", line 5 in count_nonzero
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/augmentor_utils.py", line 24 in random_flip_along_x
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/data_augmentor.py", line 93 in random_world_flip
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/data_augmentor.py", line 147 in forward
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 225 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
...
Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized
Thread 0x00007f85e9ffb700 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/pydevd.py", line 172 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85e97fa700 (most recent call first):
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 306 in wait
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 558 in wait
File "/home/fare01/.pycharm_helpers/pydev/pydevd.py", line 150 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85cffff700 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 292 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Thread 0x00007f85cf7fe700 (most recent call first):
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 306 in wait
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/queue.py", line 179 in get
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 367 in _on_run
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 218 in run
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/threading.py", line 890 in _bootstrap
File "/home/fare01/.pycharm_helpers/pydev/_pydev_bundle/pydev_monkey.py", line 795 in call
Current thread 0x00007f88cc4640c0 (most recent call first):
File "/home/fare01/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_trace_dispatch_regular.py", line 422 in call
File "/mnt/data/anaconda3/envs/fare_st3d/lib/python3.8/site-packages/numpy/core/numeric.py", line 424 in count_nonzero
File "<array_function internals>", line 5 in count_nonzero
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/augmentor_utils.py", line 24 in random_flip_along_x
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/data_augmentor.py", line 93 in random_world_flip
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/augmentor/data_augmentor.py", line 147 in forward
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 225 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/dataset.py", line 233 in prepare_data
File "/mnt/data/fare01/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 415 in getitem
...

Question about self-training on Waymo-KITTI vs Waymo-Nuscenes

Hi Jihan,

With ST3D can we do the self-training on Waymo-KITTI and use that same model checkpoint when testing on nuScenes as the target? Or did you have to re-train the waymo model specifically for the nuScenes as target domain to get the Waymo-nuScenes checkpoint?

From my understanding of your paper, because you generate pseudo labels for training on the target domain, the model would have to be re-trained for every new target domain. Am I correct in understanding this?

Again, thanks for your work and prompt responses. I appreciate your help.

Which Nuscenes dataset should I use ?

there are many kind of Nuscenes datasets, but i can't find the one that fit the dataset struct you mentioned in Readme.
I am looking for datasets from https://www.nuscenes.org/download#

The performance of pre-train with ROS on PVRCNN from Waymo->KITTI

I got 19.27 for moderate AP3D with IOU threshold 0.7 on car category on PVRCNN with ROS from Waymo->KITTI, which is worse than the performance of that on PVRCNN without ROS shown in the paper(22.01). Then I trained the PVRCNN without ROS on my machine, and got the best result over 22 on epoch. I used the original config on cfgs/da-waymo-kitti_models/pvrcnn/ pvrcnn_old_anchor.yaml and cfgs/da-waymo-kitti_models/pvrcnn/ pvrcnn_old_anchor_ros.yaml, did I do something wrong? And did you also get the unstable result on Waymo->KITTI?

Support for OpenPCDet v0.3.0

Hi,

I'm very interested in your code. Will you consider supporting OpenPCDet v0.3.0? Seems that my GPU (Nvidia RTX 3080) only supports CUDA 11 for now, and I've been kept having all kinds of issues with running CUDA 9.0 on my computer. It seems that I can only run OpenPCDet v0.3.0 with CUDA 11.

Thank you

rcnn_iou dimension issue using align_corners=True

Thanks for the great work!
I was able to install all dependencies of ST3D on RTX 3090 using the following guide
installation_guide_ST3D.txt

When I tried to reproduce results using the model provided (model-zoo),

python3 test.py --cfg_file cfgs/da-nuscenes-kitti_models/secondiou_st3d/secondiou_st3d.yaml --ckpt ../checkpoints/secondiou_st3d_ckpt.pth

I got the following results
Car [email protected], 0.70, 0.70:
bbox AP:83.3016, 66.6522, 64.9811
bev AP:80.5374, 66.3496, 64.2153
3d AP:62.5984, 48.3767, 45.5974
aos AP:83.05, 66.16, 64.30

I ran all other models and got accuracies less than what you have mentioned in Table 1.0
After many trials and training even, I got some fixes. The issue is updated PyTorch behavior for the affine grid and grid sampling.
The default behavior up to version 1.2.0 was align_corners = True. After that, the default behavior has been changed to align_corners = False, in order to bring it in line with the default for interpolate().

applying this change in ST3D/pcdet/models/roi_heads/second_head.py

grid = nn.functional.affine_grid(theta,
torch.Size((rois.size(1), spatial_features_2d.size(1), grid_size, grid_size)), align_corners=True)

pooled_features = nn.functional.grid_sample(
spatial_features_2d[b_id].unsqueeze(0).expand(rois.size(1), spatial_features_2d.size(1), height, width),
grid, align_corners=True)

gives me same results as provided in Table 1.0
Car [email protected], 0.70, 0.70:
bbox AP:92.6115, 76.7241, 73.1141
bev AP:90.4426, 77.5308, 73.3251
3d AP:71.0224, 55.5878, 51.2862
aos AP:92.53, 76.36, 72.59
I am definately able to test provided models.

Question
I trained Secondiou + ROS for 80 epochs, training results/losses look fine. However, when i am trying to perform evaluation, i am getting following error
kitti_dataset.py", line 279, in generate_single_sample_dict
pred_boxes[:, 0:3] -= self.dataset_cfg.SHIFT_COOR # (37, 1, 7)
ValueError: operands could not be broadcast together with shapes (37,1,7) (3,) (37,1,7)

After debugging and tracing back, I guess I am having a dimension mismatch issue in

ST3D/pcdet/models/roi_heads/second_head.py

Line 130 in 812b644

    
           rcnn_iou = self.iou_layers(shared_features).transpose(1, 2).contiguous().squeeze(dim=1)  # (B*N, 1)

rcnn_iou dimension should be torch.Size([1600, 1]) however i am getting torch.Size([1600, 256, 1])

I am trying to debug this issue. I will be glad for any help in this regard

Hope for Update to Latest OpenPCDet version

Hi, do you plan to update this project to the latest OpenPCDet version (e.g., OpenPCDet-0.5+)?

eval_utils.py hangs at dataloader

Hi,

My test.py used to work, but I once pressed Ctrl+C during testing, and then when I run test.py again, the program hangs on this line (line 54 in eval_utils.py):
for i, batch_dict in enumerate(dataloader):

I really don't know where the deadlock was created. Lowering num_workers does not help.

The command I ran was:
python test.py --cfg_file ./cfgs/da-nuscenes-kitti_models/pvrcnn/pvrcnn_old_anchor.yaml --batch_size 4 --ckpt ./cfgs/da-nuscenes-kitti_models/pvrcnn/ckpt_pvrcnn_source_only.pth

And it hangs at this step (epoch 0 will never proceed):
eval: 0%| | 0/943 [00:00<?, ?it/s]

When I press Ctrl+C, it says so:

[2021-10-04 17:02:27,446 eval_utils.py 40 INFO] *************** EPOCH 3 EVALUATION *****************
eval: 0%| | 0/943 [00:00<?, ?it/s]^CTraceback (most recent call last):
File "test.py", line 214, in
main()
File "test.py", line 210, in main
epoch_id, dist_test=dist_test)
File "test.py", line 64, in eval_single_ckpt
result_dir=eval_output_dir, save_to_file=args.save_to_file, args=args
File "/home/wangweijia/Desktop/UDA/ST3D/tools/eval_utils/eval_utils.py", line 54, in eval_one_epoch
for i, batch_dict in enumerate(dataloader):
File "/home/wangweijia/anaconda3/envs/st3d/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 435, in next
data = self._next_data()
File "/home/wangweijia/anaconda3/envs/st3d/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1068, in _next_data
idx, data = self._get_data()
File "/home/wangweijia/anaconda3/envs/st3d/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1024, in _get_data
success, data = self._try_get_data()
File "/home/wangweijia/anaconda3/envs/st3d/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 872, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/home/wangweijia/anaconda3/envs/st3d/lib/python3.6/queue.py", line 173, in get
self.not_empty.wait(remaining)
File "/home/wangweijia/anaconda3/envs/st3d/lib/python3.6/threading.py", line 299, in wait
gotit = waiter.acquire(True, timeout)
KeyboardInterrupt
eval: 0%|

I believe a deadlock is created somewhere, can you offer some help? I really appreciate it!

Question about ST3D with multi-category

First of all, thank you very much for opening up this work. The current models only support the category of car, and then I try to modify the configuration file to make the model support multi-category (Car, Pedestrian, Cyclist), but the current performance of Ped and Cyc is very poor(AP close to zero), I guess that there are some mistakes param in my yaml file, have you done similar work before or could you give me some suggestion? Look forward to your reply, thank u.

SN Training

Hi,

In the original paper for SN, the authors use SN in a fine-tuning process with a detector that has already been pre-trained on the source-to-target task. From looking at the code, the SN implementation in this repo seems to be training from scratch with SN in the data augmentation process - am I right in this understanding?

Is there a difference in results if you use SN for a train-from-scratch model vs using a pre-trained network and fine-tuning a few more epochs with the SN config?

How to use and visualise the test results

Hi Jihan,

I'm very new to this field so please bear with me...

I've tested your code for nuScenes -> KITTI task using SECOND-IoU+ST3D, here is my command:
python test.py --cfg_file ./cfgs/da-nuscenes-kitti_models/secondiou_st3d/secondiou_st3d.yaml --batch_size 4 --ckpt ./cfgs/da-nuscenes-kitti_models/secondiou_st3d/ckpt_secondiou_st3d.pth

And the output is this:
Car [email protected], 0.70, 0.70: bbox AP:80.3036, 65.9065, 64.7457 bev AP:78.9386, 65.4960, 64.0457 3d AP:63.2802, 50.1199, 47.2038 aos AP:80.08, 65.47, 64.13 Car [email protected], 0.70, 0.70: bbox AP:83.3016, 66.6522, 64.9811 bev AP:80.5374, 66.3496, 64.2153 3d AP:62.5984, 48.3767, 45.5974 aos AP:83.05, 66.16, 64.30 Car [email protected], 0.50, 0.50: bbox AP:80.3036, 65.9065, 64.7457 bev AP:85.6287, 72.1740, 71.4272 3d AP:85.3973, 71.3382, 70.1702 aos AP:80.08, 65.47, 64.13 Car [email protected], 0.50, 0.50: bbox AP:83.3016, 66.6522, 64.9811 bev AP:87.4123, 73.7019, 72.3442 3d AP:87.1241, 72.1693, 70.6485 aos AP:83.05, 66.16, 64.30

But I have difficulties in understanding this. I know the results are saved as result.pkl and some logs .txt files, but I really don't know their meanings and how to use them. Also, is there a way to visualise the output, e.g. drawing the predicted & ground truth bounding boxes on the images and LiDAR point clouds?

Sincerely,
Wei

Anchor Size Setting

For the WAYMO->KITTI adaptation, the size normalized model uses the custom object size for anchors (somewhere in between WAYMO-KITTI) even though the training data object size is reduced to KITTI and the test is the KITTI itself. But from a theoretical point of view, it should be used as KITTI mean object size in secondiou/secondiou_old_anchor_sn.yaml.
Would like to ask how does the author decides on those sizes or not to use the KITTI default?

When the code will be released?

Hi,

I am very interested to your work. I wonder when you plan to release the main part of the code. Thanks.

Regards,

Config file for Waymo to Nuscenes

Hello,
Would it be possible to have the config file for the SECOND-iou with Statistical Normalization for the Waymo to Nuscenes scenario?

Thank you

Config files for NuScenes to KITTI domain adaptation

Hi,
Will you make the DA config files for the NuScenes to KITTI problem available?

Thanks

Model pths for ST3D (no SN) on waymo-nuscenes, waymo-kitti

Hi Jihan,

Thanks for releasing all your checkpoint files on the model zoo. I'd like to do comparisons with some configurations you've used in your paper.

Could I get the SECOND-IoU and PV-RCNN pth files for waymo-nuscenes and waymo-kitti specifically with the ST3D (no SN) configuration?

The target domain for pre-training stage

Hi,
Thanks for your great work! I see DATA_CONFIG_TAR in yaml file and I wonder if the pre-training stage use any information in target domain?

web cam integration

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

	gt_infos = {
	'gt_boxes': gt_box,
	'cls_scores': pred_cls_scores,
	'iou_scores': pred_iou_scores,
	'memory_counter': np.zeros(gt_box.shape[0])
	}