iceory / pmf Goto Github PK

View Code? Open in Web Editor NEW

184.0 184.0 36.0 1.04 MB

Perception-aware multi-sensor fusion for 3D LiDAR semantic segmentation (ICCV 2021)

License: MIT License

Python 99.82% Shell 0.18%

pmf's People

Contributors

Stargazers

Watchers

pmf's Issues

SemanticKITTI Validation Set Results

Hello, thank you for publishing this repository, your work is very insightful.

Regarding Table 1. in the paper about validation set comparisons on SemanticKITTI, may I ask how you obtained the mIOU results of the other methods? I know that you evaluate on your own dataset created by create_fov_dataset.py , but do you:

Use the published pretrained checkpoint from those methods (e.g. Salsanext, Cylinder3D, etc.) to evaluate directly even though most of them are trained on the entire point cloud (360 degree fov). Or,
Use the published model architecture and training settings of those methods and train yourselves using the limited fov dataset

If none of the above is correct, it would be much appreciated if you give me some insights on how you conducted the comparison. Thanks again for your great work!

Can anyone tell me where the various configuration version numbers are（like pytorch,torchvision）？

key Erroe：tr

请问在pc_pro--dataset--semantic_kitti--parser中的第203行出现的calib_all字典中并没有出现名为Trd1key,这一问题应该怎么解决呢？

Can you provide some codes related to visualization? thank you very much indeed

questiones about the model performance

Hello, thanks you for sharing the code.

I have run the pmf model in semantic kitti training set using the default settings. However, the evaluation result in validation set is lower than the value 63.9 reported in the paper.

Meanwhile, I also run the salsanext model provied in the codebase in semantic kitti training set using the default settings. The evaluation result in validation set is higer than the value 59.4 reported in the paper and the pmf model.

Could you give some suggestions?

在验证时报错：stack expects each tensor to be equal size, but got [19961] at entry 0 and [123389] at entry 1

具体情况如下;
===init env success===
Traceback (most recent call last):
File "/home/lcy/桌面/PMF-master/tasks/pmf_eval_semantickitti/infer.py", line 369, in
exp.run()
File "/home/lcy/桌面/PMF-master/tasks/pmf_eval_semantickitti/infer.py", line 354, in run
self.inference.run()
File "/home/lcy/桌面/PMF-master/tasks/pmf_eval_semantickitti/infer.py", line 134, in run
self.evaluator.addBatch(
File "/home/lcy/桌面/PMF-master/tasks/pmf_eval_semantickitti/../../pc_processor/metrics/iou_eval.py", line 44, in addBatch
idxs = torch.stack([x_row, y_row], dim=0)
RuntimeError: stack expects each tensor to be equal size, but got [19961] at entry 0 and [123389] at entry 1

Process finished with exit code 1

请问该如何修改呢？

data placement structure of the dataset?

Dear author
what is the data placement structure of the dataset?
such as seamantic kitti, like, are there images and lidar data under sequence00?

corresponding images dataset download link

Hello
Dear author, I only found the lidar data of the semanti kitti dataset, can I get its corresponding images dataset download link?
or Nuscenes，...
Sorry, I really can't find it.

Thanks!

UnboundLocalError: local variable 'mean_acc' referenced before assignment

when i run PMF code on Sensat Urban Dataset，it show that “UnboundLocalError: local variable 'mean_acc' referenced before assignment” and "UnboundLocalError: local variable 'lr' referenced before assignment"。How can I solve this problem？
Complete error reporting information is as follows

/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  FutureWarning,
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
| distributed init (rank 1): env://
| distributed init (rank 2): env://
| distributed init (rank 0): env://
>> Init a recoder at  ../../../experiments/PMF-sensat/log_SensatUrban_PMFNet-resnet101_bs12-lr0.001_baseline-timestamp
loading data frame...
0it [00:00, ?it/s]
Using 0 data frame from train split
loading data frame...
0it [00:00, ?it/s]
Using 0 data frame from val split
Generate 0 samples from train split
Generate 0 samples from val split
loading data frame...
0it [00:00, ?it/s]
Using 0 data frame from train split
loading data frame...
0it [00:00, ?it/s]
Using 0 data frame from val split
Generate 0 samples from train split
Generate 0 samples from val split
focal_loss alpha: [ 0.   1.   1.   1.   2.   2.5  1.   3.   1.   1.   1.   1.  10.   2.5]
loading data frame...
0it [00:00, ?it/s]
Using 0 data frame from train split
loading data frame...
0it [00:00, ?it/s]
Using 0 data frame from val split
Generate 0 samples from train split
Generate 0 samples from val split
[IOU EVAL] IGNORE:  tensor([0])
[IOU EVAL] INCLUDE:  tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13])
[IOU EVAL] IGNORE:  tensor([0])
[IOU EVAL] INCLUDE:  tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13])
/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
===init env success===
[IOU EVAL] IGNORE:  tensor([0])
[IOU EVAL] INCLUDE:  tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13])
[IOU EVAL] IGNORE:  tensor([0])
[IOU EVAL] INCLUDE:  tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13])
/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
[IOU EVAL] IGNORE:  ===init env success===
tensor([0])
[IOU EVAL] INCLUDE:  tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13])
[IOU EVAL] IGNORE:  tensor([0])
[IOU EVAL] INCLUDE:  tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13])
/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
===init env success===
Traceback (most recent call last):
  File "main.py", line 148, in <module>
    exp.run()
  File "main.py", line 99, in run
    self.trainer.run(epoch, mode="Train")
  File "/home/yczhou/PMF-master/tasks/sensat_urban/pmf/trainer.py", line 528, in run
    "Acc": mean_acc.item(),
UnboundLocalError: local variable 'mean_acc' referenced before assignment
Traceback (most recent call last):
  File "main.py", line 148, in <module>
    exp.run()
  File "main.py", line 99, in run
    self.trainer.run(epoch, mode="Train")
  File "/home/yczhou/PMF-master/tasks/sensat_urban/pmf/trainer.py", line 528, in run
    "Acc": mean_acc.item(),
UnboundLocalError: local variable 'mean_acc' referenced before assignment
Traceback (most recent call last):
  File "main.py", line 148, in <module>
    exp.run()
  File "main.py", line 99, in run
    self.trainer.run(epoch, mode="Train")
  File "/home/yczhou/PMF-master/tasks/sensat_urban/pmf/trainer.py", line 448, in run
    tag="{}_lr".format(mode), scalar_value=lr, global_step=epoch)
UnboundLocalError: local variable 'lr' referenced before assignment
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 48867) of binary: /home/yczhou/anaconda3/envs/buct-bishe/bin/python
Traceback (most recent call last):
  File "/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/site-packages/torch/distributed/run.py", line 718, in run
    )(*cmd_args)
  File "/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/yczhou/anaconda3/envs/buct-bishe/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent
    failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
main.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2022-04-15_18:58:42
  host      : zkti
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 48868)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2022-04-15_18:58:42
  host      : zkti
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 48869)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-04-15_18:58:42
  host      : zkti
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 48867)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

only one element tensors can be converted to Python scalars

Thank you very much for your work， and I am getting errors like this during training：
only one element tensors can be converted to Python scalars

It can be trained normally at the beginning, but this error is reported after a few hundred iters. Have you ever met? HELP

Is Pretrained Checkpoint Available?

Hi @iris0329 @ICEORY, thanks for open-sourcing your work!

Regarding the reported results, could you share with me the pretrained checkpoint for reproduction? Thanks a lot for your help!

如何将这个项目跑起来？

when PMF is applied to SenSatUrban，which paper is your BEV based on?And how the image data are made? A top view of a point cloud？Please elaborate. Thank you！

pretrained model

Hi! Your paper and code are fantastic! I'm wondering if you could release the pretrained model for us to download and test. Thank you!

Request source code or results of image-only adversarial analysis

Hello author, thank you for your contribution to the community! I would like to ask you how to implement adversarial analysis of camera-only FCN based methods in ablation experiments, I am strongly curious. In my work, I want to implement it, if possible, I hope you can share the source code of the relevant part, or would you like to provide the prediction visualization results of 002777.png of the validation set 08? As shown below:

How to assign semantic prediction values to points projected beyond the images

Hi!

One question really haunts me about the experimental results of nuScenes. How to assign semantic prediction values to points projected beyond the images?

Looking forward to your reply.

PMF只能做能投影到图像区域的点云分割？

PMF-ResNet50 on SemanticKITTI Validation Set?

Hi~

Thanks for the open-source repo of your excellent work!

I notice that PMF-ResNet50 significantly outperforms PMF-ResNet34 on nuScenes Validation Set, and you even adopt ResNet101 on SensatUrban Test Set.

However, the result of PMF-ResNet50 (or deeper backbone) on SemanticKITTI Validation Set is unavailable. Did you try it before? Intuitively, it will also bring gains. Or did I miss something important?

about pre-trained weights

Hi authors, Could you release the pre-training weights of the model, especially trained in nuscenes, nuscenes dataset is too big for me. Thank you.

Semantic Kitti Dataset

Can someone help me with the dataset structure of Semantic Kitti and where can I find the datasets, and how many images to be used per sequence of left and right?

I am very confused

请问EPMF的代码可以分享吗？

training time

Thanks for your amazing work, and I'm care about the time of training consuming.
From the config_server_kitti.yaml, look like use 4x3090 GPU with batch size 8. Would you like to share your training time?
More, limited by hardware，do you think that can get comparable performance to the paper report using a single 3090? In addition, is it possible to use amp in torch？

Has anyone reproduced performance comparable to the results reported in the paper？

Limited by resources, I have tried training with two 3090s (bs=4*2) and using amp, but seems unable to reproduce similar performance.

UnboundLocalError: local variable 'mean_acc' referenced before assignment

About calib.txt in SemanticKITTI

Hello author, according to the read_calib(calib_path) function of pc_processor->dataset->semantc_kitti->parser in your code,

def read_calib(calib_path):
    """
    :param calib_path: Path to a calibration text file.
    :return: dict with calibration matrices.
    """
    calib_all = {}
    with open(calib_path, 'r') as f:
        for line in f.readlines():
            if line == '\n':
                break
            key, value = line.split(':', 1)
            calib_all[key] = np.array([float(x) for x in value.split()])
calib_out = {}
# 3x4 projection matrix for left camera
calib_out['P2'] = calib_all['P2'].reshape(3, 4)
calib_out['Tr'] = np.identity(4)  # 4x4 matrix
calib_out['Tr'][:3, :4] = calib_all['Tr'].reshape(3, 4)
return calib_out

Does your calib.txt file already contain the ''Tr'' value? Why do I download only '''P0''', '''P1''', '''P2'', '''''P3' from the official website?

about experiment environment

Hello, I'm sorry to disturb you. According to the paper, I only konw about you did it under the GeForce RTX 3090. But some errors occurred when I wanted to reproduce your experiment. So could you please tell me exactly what's your experiment environment? (Something like requirements.txt )

When shall we use pcd_aug?

Thanks in advance for sharing your code for public research. The code is very standardized and easy to understand.

For my understanding, ./pc_processor/dataset/preprocess/augmentor.py is to preform PC augmenting operation such as flip, translation, etc. However, my concern is whether this will break image matching ( Lidar to camera2 matrix). Then I noticed that you have set the value of pcd_aug to false in ./tasks/pmf/trainer.py.

My questions:

1. Do you adopt pcd_aug in training?

2. In what situations should we adopt pcd_aug?

Looking forward to your reply, thanks.