Hi, when I use the fusion moudles in ./experiments, I found there is no file named 'pp_train_dets.pkl' as the 'train_det' or 'val_det'. Could you please tell me how can I solve this problem? Thank you.

Hi. Thanks for your open source work!

I met a error when I exec the experiments/pp_pv_40e_dualadd_subabs_C/eval.sh and it was
FileNotFoundError: [Errno 2] No such file or directory: '/media/lzw/new_add/dataset/kitti/testing/image_02/0001/000147.png

I checked that my dataset and it seems that all pictures in /testing/image_02, like /testing/image_02/000147.png instead of /testing/image_02/0001/000147.png.

Is there some pre-processing I missed?

Hope for your reply! Thanks!

ValueError: Expected more than 1 value per channel when training, got input size [1, 512].

Please let me know the environment details.
pytorch 1.1.0, cuda10.2, and torchvision is 0.6.0 on python 3.7

Hello,

many thanks for your work and making your code public. Are my results correct? I have not change anything. I have cloned the repository, put the data under right folders and executed "pp_pv_40e_mul_C/eval.sh"

I look forward to hearing from you.

Best regards

==========================tracking evaluation summary===========================
Multiple Object Tracking Accuracy (MOTA) -0.044093
Multiple Object Tracking Precision (MOTP) 0.854040
Multiple Object Tracking Accuracy (MOTAL) 0.789259
Multiple Object Detection Accuracy (MODA) 0.789616
Multiple Object Detection Precision (MODP) 0.890653

Recall 0.891036
Precision 0.922639
F1 0.906562
False Alarm Rate 0.236567

Mostly Tracked 0.773148
Partly Tracked 0.203704
Mostly Lost 0.023148

True Positives 11342
Ignored True Positives 1616
False Positives 951
False Negatives 1387
Ignored False Negatives 1192
Missed Targets 1387
ID-switches 9265
Fragmentations 9295

Ground Truth Objects (Total) 13921
Ignored Ground Truth Objects 2808
Ground Truth Trajectories 240

Tracker Objects (Total) 13168
Ignored Tracker Objects 875
Tracker Trajectories 13130

Hi! Thanks for your great work. The evaluation using the pre trained models worked as described. However, I wanted to evaluate on a different set of detections, particularly from PointRCNN. I have grabbed detections from AB3D , where they provide detections in their /data/KITTI for car,pedestrian, etc and have changed the format to fit your .pkl format

I get the following error only on sequences 0007.txt and 0019.txt, but the cod works on the rest.
[2020-04-16 19:36:15,628][eval_seq.py][line: 90][ INFO] Test: [9/11] Sequence ID: KITTI-0019
[2020-04-16 19:36:16,832][eval_seq.py][line: 161][ INFO] Test Frame: [0/1036] Time 1.201(1.201)
Traceback (most recent call last):
File "eval_seq.py", line 205, in
main()
File "eval_seq.py", line 72, in main
validate(val_dataset, tracking_module, args.result_sha, part='val')
File "eval_seq.py", line 107, in validate
seq_loader, tracking_module)
File "eval_seq.py", line 154, in validate_seq
input[0], det_info, dets, det_split)
File "/home/david/Documents/trackers/mmMOT/tracking_model.py", line 81, in predict
assign_id, assign_bbox)
File "/home/david/Documents/trackers/mmMOT/tracking_model.py", line 136, in align_id
dets_out[i]['id'] += id_offset
RuntimeError: result type Float can't be cast to the desired output type Long

So couple of questions.

1. Is there anything particular with those sequences that makes that error only there, or is probably coming from detection file(made my own .pkl file replicating your own)
2. After deleting these sequences from detections, I get on validation set: ==========================tracking evaluation summary===========================
   Multiple Object Tracking Accuracy (MOTA) 0.416359
   Multiple Object Tracking Precision (MOTP) 0.865596
   Multiple Object Tracking Accuracy (MOTAL) 0.460030
   Multiple Object Detection Accuracy (MODA) 0.460272
   Multiple Object Detection Precision (MODP) 0.941595

Recall 0.682547
Precision 0.806635
F1 0.739421
False Alarm Rate 0.507463

2. Would you think this is somehow expected or am I missing something I have to change in your code to accept other detections? It seems a bit low, particularly knowing it was trained on KITTI and on PointClouds, so I was hoping it would not differ by much.

Thanks!

Hello! Thank you for sharing your work.
I have did the evaluation using PointPillars detection results and it aligned with yours. However, when I tried to use rrc results, I couldn't get the evaluation properly.
I was wondering if you have any idea about how to modify the code for rrc det (2D det). I checked the config file for rrc, the det type is still '3D'. I have tried changed it to 2D but the problem still exists.

i' am a beginner in deep-learning, where to input （e.g.any two pictures from kitti, including point cloud information） and get the prediction from the pretrained model.Thanks.

Hi，
I can see from Kitti's website that your multi-target tracking indicators are very good，but I meet some problems about submitting the results of multi-target tracking test set to Kitti's server. I have sent an email to your mailbox. I wonder if you have seen it
That porblem is that the data in your zip archive submitted to Kitti's server has the false format.
I have submitted the results of multi-target tracking test sets to the server twice, according to the document requirements, but the server still says that the data in your zip archive has the false format. I can't find the problems. Because I am writing a paper, I am very anxious to solve this problem. I have attached the submitted compressed file. If you are free, can you help me to see if there is any problem with my file format? Or can you provide a document in the correct format that you submitted？
thank you very much!

when converting kitti to coco i get following error
No such file or directory: '../../data/kitti_tracking//data_tracking_image_2/testing/image_02/0000'
data doesnt have it

Hi ZwwWayne,
which steps are necessary to create the fitting data structure in pp_val_dets.pkl? You said, you have used the SECOND Detector for detection. But the data structure and the results in the .pkl file have a different shape than in mmMot and pp_train_dets.pkl.
Which additional steps are necessary to import a detection result from SECOND and use it for mmMOT, if you are not using the pkl files from google drive?

Thank you :)

Hi, thanks you for so great work, I just wonder where you will give the nuscene result in the future, as your paper's result is so low in other two paper <> and <<Parallelized 3D Tracking and Forecasting with Graph Neural Networks and Diversity Sampling>>, AMOTA is only 2.13 on nuscene.

Thanks for sharing your code, I wonder that whether I can choose which GPU I can train this model on or not?

hey hi,
I was trying out your work but I came to met with a missing camera calib files kitti_t_o/training/calib files can you lease let me know where to get them?

Thanks

Can we run this repo on NuScenes dataset?

Hi, I see your result in kitti, the time consuming is 0.01s , it's so excellent, I just wonder if you can me what gpu you are using?

Hi, how could I only use the image or the point cloud for tracking, I tried to disable the fusion mode in config file, but it has an error that the point feature size is 0.

Thanks for your help.

Traceback (most recent call last):
File "D:/Projects/Track/PointCloudTrack/mmMOT/main.py", line 275, in
main()
File "D:/Projects/Track/PointCloudTrack/mmMOT/main.py", line 37, in main
with open(args.config) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'cfgs/config_res50.yaml'

Process finished with exit code 1

If I need to use the pretrained model to test the results, what data will I use? What is the directory composition and if it's training, what is the directory composition? Looking forward to your reply!

Hi, thanks for your great work:)
There's some problem with understanding the official code:
curr_dets, add_num = add_miss_dets(
prev_dets,
dets,
iou_threshold=iou_threshold,
fix_threshold=fix_threshold) # add missed dets
add_count += add_num
add_frame += int(add_num > 0)

Why do we need the function of add_miss_dets? Actually the program always end up at
if len(index) == 0:
return dets, 0

inside the function, when dealing with pp_train_dets.pkl and pp_val_dets.pkl.

I want to evaluate Data with the kitti_dev kit but i received this error:

python3 evaluate_tracking.py /home/sim/mmMOT eval
Processing Result for KITTI Tracking Benchmark
Traceback (most recent call last):
File "evaluate_tracking.py", line 980, in
success = evaluate(result_sha, root, mail)
File "evaluate_tracking.py", line 914, in evaluate
e = trackingEvaluation(t_sha=result_sha, root=root, part=part, mail=mail,cls=c)
File "evaluate_tracking.py", line 135, in init
self.t_path = os.path.join(root, t_sha, part)
File "/usr/lib/python3.6/posixpath.py", line 94, in join
genericpath._check_arg_types('join', a, *p)
File "/usr/lib/python3.6/genericpath.py", line 149, in _check_arg_types
(funcname, s.class.name)) from None
TypeError: join() argument must be str or bytes, not 'Mail'

hi, I run:
python -u main.py --config ./config.yaml/config.yaml --result-path=./results
then have the error:
Traceback (most recent call last):
File "main.py", line 275, in
main()
File "main.py", line 115, in main
lr_scheduler, last_iter + 1, tb_logger)
File "main.py", line 173, in train
part=args.part)
File "main.py", line 235, in validate
step, args.result_path, part=part)
ValueError: not enough values to unpack (expected 8, got 5)

this is I can not fix! Can you help m?

Also. at the beginning, I find that "Add [0] cars in [0/3975] images". as shown as below. Is that ok?

Only calculate the last transform with weight 0.001
Building dataset using dets file ./data/pp_train_dets.pkl
Detect [ 16258] cars in [3365/3975] images
Add [0] cars in [0/3975] images
Building dataset using dets file ./data/pp_train_dets.pkl
Detect [ 16258] cars in [3365/3975] images
Add [0] cars in [0/3975] images
Building dataset using dets file ./data/pp_val_dets.pkl
Detect [ 13170] cars in [3475/3945] images
Add [0] cars in [0/3945] images

Ubuntu 18.04 Cuda compilation tools, release 9.0, V9.0.176
pytorch 1.5.0

(mmmot) lzw@resplendent-star:~/resplendent_code/3d_tracking/mmMOT-master$ python -u eval_seq.py --config /home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/config.yaml --load-path=/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/model/pp_pv_40e_dualadd_subabs_C.pth --result-path=/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/results --result_sha=eval
Fusion Module C: split sigmoid weight gated point, image fusion
use Skip Pooling in appearance model
use avg in pointnet feat
Use minus_abs similarity with fusion module
End version V2 by avg
NewEndIndicator_v2(
(conv0): Sequential(
(0): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1))
(1): GroupNorm(1, 512, eps=1e-05, affine=True)
(2): ReLU(inplace=True)
)
(conv1): Sequential(
(0): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
(1): GroupNorm(1, 512, eps=1e-05, affine=True)
(2): ReLU(inplace=True)
(3): Conv1d(512, 128, kernel_size=(1,), stride=(1,))
(4): GroupNorm(1, 128, eps=1e-05, affine=True)
(5): ReLU(inplace=True)
(6): Conv1d(128, 1, kernel_size=(1,), stride=(1,))
(7): Sigmoid()
)
)
=> loading checkpoint '/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/model/pp_pv_40e_dualadd_subabs_C.pth'
Building dataset using dets file ./data/pp_train_dets.pkl
Detect [ 16258] cars in [3365/3975] images
Add [0] cars in [0/3975] images
Building dataset using dets file ./data/pp_val_dets.pkl
Detect [ 13170] cars in [3475/3945] images
Add [0] cars in [0/3945] images
[2020-04-25 19:21:41,270][eval_seq.py][line: 64][ INFO] args: Namespace(config='/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/config.yaml', evaluate=False, load_path='/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/model/pp_pv_40e_dualadd_subabs_C.pth', memory=False, recover=False, result_path='/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/results', result_sha='eval')
[2020-04-25 19:21:41,271][eval_seq.py][line: 65][ INFO] config: {'augmentation': {'input_size': 224, 'test_resize': 224},
'batch_size': 1,
'det_type': '3D',
'dropblock': 0,
'fixed_wd': True,
'gt_det_ratio': 0,
'loss': {'det_loss': 'bce',
'det_ratio': 1.5,
'link_loss': 'l2',
'smooth_ratio': 0,
'trans_last': True,
'trans_ratio': 0.001},
'lr_scheduler': {'base_lr': 0.0003,
'div_factor': 10.0,
'lr_max': 0.0006,
'max_iter': 134200,
'moms': [0.95, 0.85],
'optim': 'Adam',
'pct_start': 0.4,
'type': 'one_cycle'},
'model': {'affinity_op': 'minus_abs',
'appear_arch': 'vgg',
'appear_fpn': False,
'appear_len': 512,
'appear_skippool': True,
'end_arch': 'v2',
'end_mode': 'avg',
'neg_threshold': 0.2,
'point_arch': 'v1',
'point_len': 512,
'score_arch': 'branch_cls',
'score_fusion_arch': 'C',
'softmax_mode': 'dual_add',
'test_mode': 2},
'momentum': 0.9,
'print_freq': 100,
'sample_max_len': 2,
'save_path': '/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C',
'tracker_type': '3D',
'train_det': './data/pp_train_dets.pkl',
'train_fix_count': 0,
'train_fix_iou': 1,
'train_link': './data/train.txt',
'train_root': './kitti_t_o/training',
'train_source': './kitti_t_o/training/',
'use_dropout': False,
'use_frustum': False,
'use_moving_average': False,
'val_det': './data/pp_val_dets.pkl',
'val_fix_count': 0,
'val_fix_iou': 1,
'val_freq': 3355,
'val_link': './data/val.txt',
'val_root': './kitti_t_o/training',
'val_source': './kitti_t_o/training/',
'weight_decay': 0.01,
'without_reflectivity': True,
'workers': 1}
[2020-04-25 19:21:41,271][eval_seq.py][line: 69][ INFO] Evaluation on traing set:
[2020-04-25 19:21:41,271][eval_seq.py][line: 89][ INFO] Test: [0/10] Sequence ID: KITTI-0003
Traceback (most recent call last):
File "eval_seq.py", line 204, in
main()
File "eval_seq.py", line 70, in main
validate(train_dataset, tracking_module, args.result_sha, part='train')
File "eval_seq.py", line 106, in validate
seq_loader, tracking_module)
File "eval_seq.py", line 153, in validate_seq
input[0], det_info, dets, det_split)
File "/home/lzw/resplendent_code/3d_tracking/mmMOT-master/tracking_model.py", line 70, in predict
det_imgs, det_info, det_split)
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lzw/resplendent_code/3d_tracking/mmMOT-master/modules/tracking_net.py", line 166, in forward
feats, trans = self.feature(dets, det_info)
File "/home/lzw/resplendent_code/3d_tracking/mmMOT-master/modules/tracking_net.py", line 139, in feature
det_info['points_split'].long().squeeze(0))
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lzw/resplendent_code/3d_tracking/mmMOT-master/modules/point_net.py", line 26, in forward
x, trans = self.feat(x, point_split)
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lzw/resplendent_code/3d_tracking/mmMOT-master/modules/point_net.py", line 119, in forward
trans1 = self.stn1(x)
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lzw/resplendent_code/3d_tracking/mmMOT-master/modules/point_net.py", line 79, in forward
x = self.relu(self.fc_bn1(self.fc1(x)))
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/nn/modules/normalization.py", line 225, in forward
input, self.num_groups, self.weight, self.bias, self.eps)
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/nn/functional.py", line 1971, in group_norm
+ list(input.size()[2:]))
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/nn/functional.py", line 1902, in _verify_batch_size
raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size [1, 512]

Hello,
Again thanks for the great work. I noticed that in the evaluation results, the score is hardcoded to 0.90 in data_util.py on writte_kitti_result function. I attempted to obtain it by using frames_det[i]['score'][j].item() (line 115), but get key error. Is it possible to obtain the actual confidence score when writing result to text file?

This isnt a problem per se, as in kitti evaluation the tracking score is not used, but I would like to evaluate it on a different benchmark.

Thanks again.

Hello, thank you for your patient answer before!

I have successfully converted the detection result of pedestrian into PKL file, but this tracking result is wrong.Is there something in the source code that needs to be changed?

0 0 Pedestrian 0.0000 0.0 1.0247 432.3253 157.4274 489.2022 253.3301 0.7070 0.9905 1.8464 -2.9469 1.4038 14.2109 0.8202 0.9000
0 1 Pedestrian 0.0000 0.0 0.9821 777.8826 156.5615 852.3547 275.3155 0.7212 0.9332 1.7715 3.2351 1.4160 11.0956 1.2658 0.9000
0 2 Pedestrian 0.0000 0.0 1.0225 728.9045 163.8145 803.7483 281.6840 0.7260 0.9374 1.7016 2.3932 1.4624 10.7632 1.2413 0.9000
0 3 Pedestrian 0.0000 0.0 1.1944 274.1638 159.1454 315.4498 228.3974 0.7073 0.9570 1.7940 -8.3320 1.2424 18.8994 0.7792 0.9000
0 4 Pedestrian 0.0000 0.0 1.7148 242.2507 156.7200 273.1434 219.0968 0.6925 0.9498 1.7945 -10.2870 1.1120 20.8947 1.2573 0.9000
0 5 Pedestrian 0.0000 0.0 1.6079 212.0153 158.1557 241.7269 220.2841 0.7328 0.9579 1.7636 -11.0709 1.1309 20.6565 1.1159 0.9000
0 6 Pedestrian 0.0000 0.0 1.5004 313.2111 161.4416 343.5999 229.4428 0.7140 0.9663 1.8240 -7.6909 1.3144 19.5498 1.1256 0.9000
0 7 Pedestrian 0.0000 0.0 1.7051 267.9218 160.2859 295.9897 220.4504 0.6514 0.8825 1.7511 -9.6614 1.1643 21.0911 1.2755 0.9000
0 8 Pedestrian 0.0000 0.0 -2.9378 472.9132 170.6904 487.4966 197.8967 0.6138 0.7820 1.6724 -7.7281 1.0708 43.7771 -3.1125 0.9000
0 9 Pedestrian 0.0000 0.0 1.9468 313.2138 160.3748 328.0861 189.7832 0.5961 0.7133 1.7232 -16.8083 0.5453 41.7835 1.5643 0.9000
1 10 Pedestrian 0.0000 0.0 1.0133 734.2387 156.5722 809.4894 279.1440 0.6989 0.9568 1.7472 2.4475 1.4078 10.6399 1.2395 0.9000
1 11 Pedestrian 0.0000 0.0 1.5424 277.6817 162.4916 309.3220 230.7803 0.7401 0.9706 1.7581 -8.3251 1.2961 18.7957 1.1256 0.9000
1 12 Pedestrian 0.0000 0.0 0.9810 778.4866 155.6182 852.6742 274.2399 0.7056 0.9341 1.7573 3.2200 1.3904 11.0211 1.2653 0.9000
1 13 Pedestrian 0.0000 0.0 1.7449 240.7054 157.0498 272.3944 220.6514 0.6812 0.9612 1.8190 -10.2612 1.1499 20.7737 1.2862 0.9000
1 14 Pedestrian 0.0000 0.0 1.5922 214.3796 160.5008 242.9699 221.9573 0.7155 0.9412 1.7351 -10.9561 1.1720 20.5382 1.1023 0.9000
1 15 Pedestrian 0.0000 0.0 1.3892 311.4415 160.1375 347.0051 224.6600 0.7543 0.9217 1.7188 -7.6166 1.1779 19.4202 1.0155 0.9000
1 16 Pedestrian 0.0000 0.0 1.0540 440.1449 163.6088 493.6551 255.3984 0.6731 0.9322 1.7569 -2.8023 1.4352 14.1009 0.8579 0.9000
1 17 Pedestrian 0.0000 0.0 1.6448 297.0058 163.1647 326.1418 216.2601 0.7834 0.9115 1.6285 -9.2505 1.0984 22.2369 1.2506 0.9000
1 18 Pedestrian 0.0000 0.0 0.1524 473.6938 170.2933 487.8574 197.7881 0.6059 0.7829 1.6928 -7.7032 1.0657 43.8370 -0.0214 0.9000
2 19 Pedestrian 0.0000 0.0 1.0057 736.5580 154.9566 813.0560 280.2099 0.7090 0.9428 1.7642 2.4621 1.4061 10.5158 1.2356 0.9000
2 20 Pedestrian 0.0000 0.0 1.3712 274.7328 157.9558 311.4357 229.7064 0.7045 0.9753 1.8319 -8.2704 1.2579 18.6487 0.9537 0.9000
2 21 Pedestrian 0.0000 0.0 0.9900 785.0606 160.0873 860.4614 276.9887 0.7340 0.9031 1.7150 3.2961 1.4172 10.9045 1.2834 0.9000
2 22 Pedestrian 0.0000 0.0 0.8202 441.1056 160.1876 500.1748 253.9441 0.6765 0.9477 1.7715 -2.6862 1.3893 13.9067 0.6293 0.9000
2 23 Pedestrian 0.0000 0.0 1.7226 241.9744 158.0480 273.4872 219.7649 0.6930 0.9386 1.7517 -10.1488 1.1160 20.6147 1.2650 0.9000
2 24 Pedestrian 0.0000 0.0 1.6128 213.2256 158.0890 242.1599 220.6907 0.6985 0.9279 1.7514 -10.8819 1.1260 20.3451 1.1215 0.9000
2 25 Pedestrian 0.0000 0.0 1.3686 310.7049 159.9707 345.0765 226.7844 0.6995 0.9246 1.7677 -7.5982 1.2261 19.2794 0.9931 0.9000
2 26 Pedestrian 0.0000 0.0 -2.9963 371.5776 162.4875 385.4744 186.6119 0.6519 0.8278 1.6501 -15.6088 0.4194 48.7466 -3.3063 0.9000
3 27 Pedestrian 0.0000 0.0 1.1661 795.2978 159.0406 862.1971 277.1657 0.6903 0.8523 1.7266 3.3511 1.4146 10.7894 1.4674 0.9000
3 28 Pedestrian 0.0000 0.0 0.8489 738.3781 160.8705 824.2325 288.4802 0.7534 0.9918 1.7625 2.5251 1.4930 10.3744 1.0877 0.9000
3 29 Pedestrian 0.0000 0.0 -2.1335 449.1101 163.7963 505.7742 257.9192 0.6785 0.9485 1.7487 -2.5217 1.4400 13.7102 -2.3153 0.9000
3 30 Pedestrian 0.0000 0.0 1.7950 210.7236 158.4173 245.0459 221.5185 0.6805 0.9320 1.7620 -10.8390 1.1470 20.2775 1.3042 0.9000
3 31 Pedestrian 0.0000 0.0 1.1833 312.5537 160.1314 349.8712 228.4238 0.6579 0.8814 1.7986 -7.4600 1.2637 19.1621 0.8122 0.9000
3 32 Pedestrian 0.0000 0.0 1.4749 278.9051 155.7254 311.6616 227.3993 0.6919 0.9817 1.8150 -8.1470 1.1892 18.4968 1.0601 0.9000
3 33 Pedestrian 0.0000 0.0 1.7562 239.1889 155.2683 273.0674 219.9292 0.7213 0.9409 1.8233 -10.1295 1.1134 20.4822 1.2970 0.9000
4 34 Pedestrian 0.0000 0.0 0.9731 792.6205 154.6197 871.2396 278.2610 0.7040 0.9722 1.7709 3.3654 1.4018 10.6887 1.2781 0.9000
4 35 Pedestrian 0.0000 0.0 0.8315 454.1628 158.7905 514.7567 255.1793 0.6822 0.9698 1.7921 -2.3816 1.3901 13.7074 0.6595 0.9000
4 36 Pedestrian 0.0000 0.0 1.8299 208.4269 158.2299 245.1361 222.7060 0.7072 0.9289 1.7834 -10.7671 1.1690 20.0850 1.3378 0.9000
4 37 Pedestrian 0.0000 0.0 1.1705 313.7086 158.4622 352.9647 228.2024 0.6815 0.9242 1.8262 -7.3706 1.2507 19.0787 0.8018 0.9000
4 38 Pedestrian 0.0000 0.0 0.6928 743.8461 161.7585 830.0995 288.1803 0.7516 0.9532 1.7289 2.5877 1.4743 10.2711 0.9396 0.9000
4 39 Pedestrian 0.0000 0.0 1.9524 235.5390 155.5995 273.1488 220.5753 0.6649 0.9481 1.8217 -10.0994 1.1251 20.3172 1.4910 0.9000
4 40 Pedestrian 0.0000 0.0 1.2695 274.2983 155.7473 312.7466 229.0676 0.6640 0.9387 1.8461 -8.1347 1.2245 18.3693 0.8526 0.9000
4 41 Pedestrian 0.0000 0.0 1.2511 333.4920 162.2618 365.0920 221.5257 0.7120 0.8712 1.8491 -8.2158 1.2816 22.6187 0.9027 0.9000
5 42 Pedestrian 0.0000 0.0 0.9376 799.8128 163.2561 881.0459 285.6137 0.7391 0.9337 1.7302 3.4496 1.4881 10.5521 1.2536 0.9000
5 43 Pedestrian 0.0000 0.0 1.4190 317.0211 161.0058 351.1756 230.2760 0.7167 0.9516 1.7983 -7.3005 1.2938 18.9409 1.0511 0.9000
5 44 Pedestrian 0.0000 0.0 0.6900 753.1007 158.8184 839.1979 284.2406 0.7629 0.9161 1.6941 2.6862 1.4029 10.1399 0.9489 0.9000
5 45 Pedestrian 0.0000 0.0 0.9991 459.7694 159.6794 518.5210 257.3993 0.7100 0.9602 1.8028 -2.2823 1.4204 13.6335 0.8332 0.9000
5 46 Pedestrian 0.0000 0.0 1.8138 207.1755 155.8818 245.0310 220.8359 0.7371 0.9541 1.7851 -10.7294 1.1101 19.9801 1.3210 0.9000
......

Looking forward to your reply!Good luck in your work!

Hi ,
in your paper you wrote, that you discard all detections with a score below 0.3. Is this value a parameter? I can't find it in the Code

why is so odd,what wrong with kitti dataset?

runing command:

(mmmot) lzw@resplendent-star:~/resplendent_code/3d_tracking/mmMOT-master$ python -u eval_seq.py --config /home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/config.yaml --load-path=/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/model/pp_pv_40e_dualadd_subabs_C.pth --result-path=/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/results --result_sha=eval
Fusion Module C: split sigmoid weight gated point, image fusion
use Skip Pooling in appearance model
use avg in pointnet feat
Use minus_abs similarity with fusion module
End version V2 by avg
NewEndIndicator_v2(
(conv0): Sequential(
(0): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1))
(1): GroupNorm(1, 512, eps=1e-05, affine=True)
(2): ReLU(inplace=True)
)
(conv1): Sequential(
(0): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
(1): GroupNorm(1, 512, eps=1e-05, affine=True)
(2): ReLU(inplace=True)
(3): Conv1d(512, 128, kernel_size=(1,), stride=(1,))
(4): GroupNorm(1, 128, eps=1e-05, affine=True)
(5): ReLU(inplace=True)
(6): Conv1d(128, 1, kernel_size=(1,), stride=(1,))
(7): Sigmoid()
)
)
=> loading checkpoint '/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/model/pp_pv_40e_dualadd_subabs_C.pth'
Building dataset using dets file ./data/pp_train_dets.pkl
Detect [ 16258] cars in [3365/3975] images
Add [0] cars in [0/3975] images
Building dataset using dets file ./data/pp_val_dets.pkl
Detect [ 13170] cars in [3475/3945] images
Add [0] cars in [0/3945] images
[2020-04-27 18:20:40,697][eval_seq.py][line: 64][ INFO] args: Namespace(config='/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/config.yaml', evaluate=False, load_path='/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/model/pp_pv_40e_dualadd_subabs_C.pth', memory=False, recover=False, result_path='/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/results', result_sha='eval')
[2020-04-27 18:20:40,698][eval_seq.py][line: 65][ INFO] config: {'augmentation': {'input_size': 224, 'test_resize': 224},
'batch_size': 1,
'det_type': '3D',
'dropblock': 0,
'fixed_wd': True,
'gt_det_ratio': 0,
'loss': {'det_loss': 'bce',
'det_ratio': 1.5,
'link_loss': 'l2',
'smooth_ratio': 0,
'trans_last': True,
'trans_ratio': 0.001},
'lr_scheduler': {'base_lr': 0.0003,
'div_factor': 10.0,
'lr_max': 0.0006,
'max_iter': 134200,
'moms': [0.95, 0.85],
'optim': 'Adam',
'pct_start': 0.4,
'type': 'one_cycle'},
'model': {'affinity_op': 'minus_abs',
'appear_arch': 'vgg',
'appear_fpn': False,
'appear_len': 512,
'appear_skippool': True,
'end_arch': 'v2',
'end_mode': 'avg',
'neg_threshold': 0.2,
'point_arch': 'v1',
'point_len': 512,
'score_arch': 'branch_cls',
'score_fusion_arch': 'C',
'softmax_mode': 'dual_add',
'test_mode': 2},
'momentum': 0.9,
'print_freq': 100,
'sample_max_len': 2,
'save_path': '/home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C',
'tracker_type': '3D',
'train_det': './data/pp_train_dets.pkl',
'train_fix_count': 0,
'train_fix_iou': 1,
'train_link': './data/train.txt',
'train_root': './kitti_t_o/training',
'train_source': './kitti_t_o/training/',
'use_dropout': False,
'use_frustum': False,
'use_moving_average': False,
'val_det': './data/pp_val_dets.pkl',
'val_fix_count': 0,
'val_fix_iou': 1,
'val_freq': 3355,
'val_link': './data/val.txt',
'val_root': './kitti_t_o/training',
'val_source': './kitti_t_o/training/',
'weight_decay': 0.01,
'without_reflectivity': True,
'workers': 1}
[2020-04-27 18:20:40,699][eval_seq.py][line: 69][ INFO] Evaluation on traing set:
[2020-04-27 18:20:40,699][eval_seq.py][line: 89][ INFO] Test: [0/10] Sequence ID: KITTI-0003
[2020-04-27 18:20:42,881][eval_seq.py][line: 160][ INFO] Test Frame: [0/143] Time 2.182(2.182)
[2020-04-27 18:20:59,636][eval_seq.py][line: 160][ INFO] Test Frame: [100/143] Time 0.221(0.187)
Make directory: /home/lzw/resplendent_code/3d_tracking/mmMOT-master/experiments/pp_pv_40e_dualadd_subabs_C/results/eval/train
[2020-04-27 18:21:03,890][eval_seq.py][line: 89][ INFO] Test: [1/10] Sequence ID: KITTI-0001
[2020-04-27 18:21:05,347][eval_seq.py][line: 160][ INFO] Test Frame: [0/422] Time 1.456(1.456)
[2020-04-27 18:21:40,783][eval_seq.py][line: 160][ INFO] Test Frame: [100/422] Time 0.263(0.365)
Traceback (most recent call last):
File "eval_seq.py", line 204, in
main()
File "eval_seq.py", line 70, in main
validate(train_dataset, tracking_module, args.result_sha, part='train')
File "eval_seq.py", line 106, in validate
seq_loader, tracking_module)
File "eval_seq.py", line 144, in validate_seq
for i, (input, det_info, dets, det_split) in enumerate(val_loader):
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 819, in next
return self._process_data(data)
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/_utils.py", line 369, in reraise
raise self.exc_type(msg)
FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/lzw/resplendent_code/3d_tracking/mmMOT-master/dataset/test_seq_dataset.py", line 139, in getitem
return self._generate_img_lidar(idx)
File "/home/lzw/resplendent_code/3d_tracking/mmMOT-master/dataset/test_seq_dataset.py", line 191, in _generate_img_lidar
img = Image.open(path)
File "/home/lzw/anaconda3/envs/mmmot/lib/python3.7/site-packages/PIL/Image.py", line 2770, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/media/lzw/new_add/dataset/kitti/testing/image_02/0001/000147.png'

(mmmot) lzw@resplendent-star:~/resplendent_code/3d_tracking/mmMOT-master$

it includes data_tracking_label_2.zip， data_tracking_velodyne.zip， data_tracking_image_2.zip，data_tracking_calib.zip？ thank you！

Hi @ZwwWayne,

Do you have the demo code to generate some graphical results? like those images in Figure 5 in your paper.

hi@ZwwWayne :
After evaluating the pre-training model, the TXT file corresponding to each video will be generated. How can I visualize these results? I don't seem to see any code that can actually be visualized. Can you provide the code? Thank you very much!

Hello, Do you have any detection results of second 1.6 used for your tracking algorithm, I need to install second 1.6?

I am not able to find the supplementary paper link online. Can you please share the supplementary paper?

Hi, thanks for sharing your great work, but I got a problem when I use rrc detector to train the model, the error is as below:

Det ratio 1.5
Only calculate the last transform with weight 0.001
Building dataset using dets file ./data/RRC_Detections_mat/train
Detect [0] in [0/3999] images with detections
Add [0] cars in [0/3999] images
Building dataset using dets file ./data/RRC_Detections_mat/train
Detect [0] in [0/3999] images with detections
Add [0] cars in [0/3999] images
Building dataset using dets file ./data/RRC_Detections_mat/train
Detect [0] in [0/3864] images with detections
Add [0] cars in [0/3864] images
Traceback (most recent call last):
File "/home/chencan/Phd_Projects/01_tracking/mmMOT/main.py", line 275, in
main()
File "/home/chencan/Phd_Projects/01_tracking/mmMOT/main.py", line 89, in main
last_iter=last_iter)
File "/home/chencan/Phd_Projects/01_tracking/mmMOT/utils/train_util.py", line 116, in init
self.indices = self.gen_new_list()
File "/home/chencan/Phd_Projects/01_tracking/mmMOT/utils/train_util.py", line 133, in gen_new_list
num_repeat = (all_size - 1) // origin_indices.shape[0] + 1
ZeroDivisionError: integer division or modulo by zero

I checked the code that the problem comes from the unavailable path "point_path = f'{root_dir}/velodyne_reduced/{seq_id}-{frame_id}.bin', which leads to empty detect results."

So I just wonder if I missed some important steps for data preprocessing or it is a bug.

Many thanks for your help.

Hey! Thanks for your work. Glad you've decided to use pytorch.
But looking through paper and code was not clear for me if there is any plans on utilizing the capabilities of 360 degree FOV systems like nuScenes dataset? Or maybe I've just missed the part where multiple cameras are implicit to be used. I'll be very glad for your clarification.

Also it's very interesting to know if there was any experiments with radar data.

Hello,

You provide 4 pre trained model for PointPillar. However, for RRC, I do not see the checkpoint: ckpt_best.pth.tar that we are supposed to load from the script in experiments/rrc_pfv_40e_subabs_dualadd_C/eval.sh in in your google drive, therefore it is not found when running the code for RRC detections.

Do we have to train it ourselves? Evaluation with PP has been without any issues.

Hello, first of all, thank you for your reply!I would like to ask, because the result of my pedestrian detection is a file in. TXT format, do you have the code to convert. TXT file into PKL file?I'd like to refer to it.If there is no code, can you give me a rough idea of the transformation?Thank you again!Wish you all the best in your work!

Since we use your pp_train_dets.pkl and pp_val_dets.pkl, we really need pp_test_dets.pkl generated by the exactly same pillatpoint model.

Can you release this result?

Thanks a lot!

Hello @ZwwWayne . In the proposed architecture we are fusing the lidar point cloud and the camera information .

Can you please comment on what will be the output confidence value and the bounding box coordinates ??

1. Do we take the bounding box and confidence information from lidar point cloud 3D detections (or) Camera Image 3D detections ??

2. What if the one of the sensor fails. How we get the bounding box and the confidence score information. ??

3. The output from the architecture is (y true, y new, y end , y link). How we are getting the track id's from these outputs ??

4. What is y new and y end ( Start and end of the trajectory ) ??
   Thanks In Advance ....

Hello!Thank you for your great code！I want to study pedestrian tracking. Could you please give me the detection result of pedestrian of the PointPiller model in PKL format ?Thank you. Good luck with your work!

hello，do you plan to realease your coda? thanks !

Hi, thanks for your great work. Could you please also share your pkl files for testing data?

Many thanks.

Could you let me know how did you visualize your results on image and 3D space?
Is there any reference codes?

when do the demo of Kitti of eval_seq.py ,I met this question:
RuntimeError: stack expects each tensor to be equal size, but got [5, 3, 1242, 1242] at entry 0 and [4, 3, 1242, 1242] at entry 1,please give me some help of ideas.