tiagocortinhal / salsanext Goto Github PK

Thanks for the great work.
I wonder why X-coordinate is around 10? I get around 0. (sequence 00-10 except 08)
mean in salsanext.yml: (12.12, 10.88, 0.23, -1.04, 0.21)
Here is my mean: (11.562, -0.107, 0.502, -1.060, 0.287)
Can you provide how you calculate the mean?

Hi,
Thanks for your great opening source of this project!

I have 3 questions about the segmentation and evaluation method in the test sequences.

1.) About the predicted value is not saved when I run Eval-Script
Now I can run Training-Script and Eval-Script using sequences 00-10, respectively.
First, when I ran the Training-Script, directories such as predictions were created in thelogs/2020-11-17-23: 08 salsanext-cpdirectory.
However, when the Eval-Script was executed, the pred directory for storing the value of prediction was created, but the contents of pred/sequences/<sequence No.>/predictions were all empty.
The run-time command is
$ ./eval.sh -d ~/Dataset/SemanticKITTI/dataset -p pred -m logs2/logs/2020-11-17-23: 08 salsanext-cp -s valid -c 30 -n salsanext.
I don't know what the problem is, so I would appreciate it if you could tell me.

2.) About labels of test sequences.
Next, I would like to run test and evaluate using the test sequences, but I do not know how to carry out the experiment because there is no labels.
Is it generated as a predicted value when I run some program? or if not necessary?
I would appreciate it if you could tell me how to run these scripts.

3.) How to use visualize.py
First, is there a way to save the results?
And then also, like Question 2, I don't know how to run it in an unlabeled test sequences.
Is the labels still something that can be generated by running some program in SalsaNext, or is the option instead of the label the value of predictions that should be generated by Eval-Script with -p?
It seems that visualize.py is not executed using the execution result of SalsaNext, so I would appreciate it if you could tell me how to use visualize.py.

I am very sorry for the amateur questions.
thank you.

Did you run your inference on inference on VPL32? If yes, could you provide with new configuration settings? use your pretrain model.

Great job, @TiagoCortinhal!
Could you please share the pre-trained model for the new uncertainty version of SalsaNext?
Thank you in advance!

Hello,

Thanks for making the code available. I have one question about the KNN post-processing that has been taken directly from the Rangenet++ paper. In line 83 of the script, we see :

# index with px, py to get ALL the pcld points
idx_list = py * W + px
unproj_unfold_k_rang = proj_unfold_k_rang[:, :, idx_list]

py is the indices of projected points in the y axis (height) and px is the indices of projected points in the x axis (width) and W is width. proj_unfold_k_rang is the im2col of projection image which is of the shape 1x49x131072 where 131072 comes from 64*2048 and 49 comes from 7 which the kernel size in the im2col process.

I don't understand why we multiply the y indices with width and then add the x indices. What is the reasoning behind the idx_list formulation?

Hi,
Sorry to bother you again!
I find that there is an operation ordering the points, label and proj_y/proj_x:

indices = np.arange(depth.shape[0])  
order = np.argsort(depth)[::-1]  
depth = depth[order]
indices = indices[order]
points = self.points[order]
remissions = self.remissions[order]

proj_y = self.proj_y[order]
proj_x = self.proj_x[order]

I am curisous about the purpose of this operation.
Looking forward to your reply !

Best,
Iris.

Based on the paper "A General Framework for Uncertainty Estimation in Deep Learning", I have questions about your inference code in user.py.

1. Should the final prediction be average of proj_output_r or proj_output2? The equation from the paper is average of T times monte-carlo predictions.
   https://github.com/Halmstad-University/SalsaNext/blob/cc8c75dc68d2607d16e2c82be61e7254e5b74a12/train/tasks/semantic/modules/user.py#L157

2. Is log_var2 sensor uncertainty? If yes, based on the equation, we just need to do average. Why did you do log_var_r.var()?
   https://github.com/Halmstad-University/SalsaNext/blob/cc8c75dc68d2607d16e2c82be61e7254e5b74a12/train/tasks/semantic/modules/user.py#L159

Thank you.

Hi~@TiagoCortinhal
When I tried to use your library to complete the experiment, I found a very strange problem:
I use 4 GPUs to train the network by modifying the batch_size in salsanext.yml.

When batch_size == 24, I found that GPU memory does not take up much memory:

|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+==
|   0  Tesla P40           On   | 00000000:02:00.0 Off |                    0 |
| N/A   47C    P0   157W / 250W |   8841MiB / 22919MiB |     75%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P40           On   | 00000000:03:00.0 Off |                    0 |
| N/A   50C    P0   160W / 250W |   8283MiB / 22919MiB |     77%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P40           On   | 00000000:83:00.0 Off |                    0 |
| N/A   47C    P0   151W / 250W |   8273MiB / 22919MiB |     56%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P40           On   | 00000000:84:00.0 Off |                    0 |
| N/A   46C    P0   146W / 250W |   8285MiB / 22919MiB |     79%      Default |
+-------------------------------+----------------------+----------------------+

When setting batch_size == 4, I found that the GPU memory is full:

-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+===
|   0  Tesla P40           On   | 00000000:02:00.0 Off |                    0 |
| N/A   42C    P0    69W / 250W |   2103MiB / 22919MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P40           On   | 00000000:03:00.0 Off |                    0 |
| N/A   43C    P0    79W / 250W |   1981MiB / 22919MiB |     60%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P40           On   | 00000000:83:00.0 Off |                    0 |
| N/A   41C    P0    90W / 250W |   1993MiB / 22919MiB |     86%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P40           On   | 00000000:84:00.0 Off |                    0 |
| N/A   40C    P0   110W / 250W |   1981MiB / 22919MiB |     73%      Default |
+-------------------------------+----------------------+----------------------+

Hope you can answer my confusion
Best wishes

Was trying to use this on windows machine by creating an powershell training script, but found out that build versions of packages are for linux. Also, there are some packages specific to linux-os only

Hi,

The performance of video in the homepage of this repository is so amazing.

Can you tell me, how could you generate such a point cloud video?

Best,
Iris

I have a question. How to use prediction tags to visualize point clouds for qualitative display, or which visualization tools are used? (That is, convert the predicted label at the bottom to the 3d point cloud view at the top)
E.g:

When running script with:
./train.sh -d "/mnt/d/side projects/KITTI_DATASET/dataset" -a "./salsanext.yml" -l "./logs"
it gives the following error:
Illegal instruction

Here is the full log:

/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  np_resource = np.dtype([("resource", np.ubyte, 1)])
----------
INTERFACE:
dataset /mnt/d/side projects/KITTI_DATASET/dataset
arch_cfg /home/moldoteck/SalsaNext/salsanext.yml
data_cfg config/labels/semantic-kitti.yaml
uncertainty False
Total of Trainable Parameters: 6.71M
log /home/moldoteck/SalsaNext/logs/logs/2021-3-28-16:10
pretrained
----------

----------

Opening arch config file /home/moldoteck/SalsaNext/salsanext.yml
Opening data config file config/labels/semantic-kitti.yaml
No pretrained directory found.
Copying files to /home/moldoteck/SalsaNext/logs/logs/2021-3-28-16:10 for further reference.
Sequences folder exists! Using sequences from /mnt/d/side projects/KITTI_DATASET/dataset/sequences
parsing seq 00
parsing seq 01
parsing seq 02
parsing seq 03
parsing seq 04
parsing seq 05
parsing seq 06
parsing seq 07
parsing seq 09
parsing seq 10
Using 19130 scans from sequences [0, 1, 2, 3, 4, 5, 6, 7, 9, 10]
Sequences folder exists! Using sequences from /mnt/d/side projects/KITTI_DATASET/dataset/sequences
parsing seq 08
Using 4071 scans from sequences [8]
Loss weights from content:  tensor([  0.0000,  22.9317, 857.5627, 715.1100, 315.9618, 356.2452, 747.6170,
        887.2239, 963.8915,   5.0051,  63.6247,   6.9002, 203.8796,   7.4802,
         13.6315,   3.7339, 142.1462,  12.6355, 259.3699, 618.9667])
Training in device:  cpu
Ignoring class  0  in IoU evaluation
[IOU EVAL] IGNORE:  tensor([0])
[IOU EVAL] INCLUDE:  tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
        19])
Illegal instruction

At trainer.py, line 384>
loss_m = criterion(torch.log(output.clamp(min=1e-8)), proj_labels) + self.ls(output, proj_labels.long())
should be replaced with
loss_m = criterion(torch.log(output.clamp(min=1e-8)), proj_labels.long()) + self.ls(output, proj_labels.long())

INTERFACE:
Dataset dataset
Config config/labels/semantic-kitti.yaml
Sequence 08
Predictions /home/s5620/SalsaNext/logs/
ignore_semantics False
ignore_safety True
offset 0

Opening config file config/labels/semantic-kitti.yaml
Sequence folder exists! Using sequence from dataset/sequences/08/velodyne
Labels folder exists! Using labels from /home/s5620/SalsaNext/logs/sequences/08/predictions
Using semantics in visualizer
To navigate:
b: back (previous scan)
n: next (next scan)
q: quit (exit program)

When I run visualize.py, I got above information but it didn't visualize any pattern
Did I do the wrong command ?

python3 visualize.py --dataset dataset --sequence 08 --predictions ~/SalsaNext/logs/ --ignore_safety

or this python script is not for visualization ?

FLAGS.mc doesn't exist. Instead we should use FLAGS.monte_carlo.

Hi

As you mentioned in issues 17

For the Number of Parameters we used the built-in functions of Pytorch like so: sum(p.numel() for p in model.parameters() if p.requires_grad)
For the FLOPs we used this package: https://github.com/sovrasov/flops-counter.pytorch
Originally posted by @TiagoCortinhal in #17 (comment)

Following your advice, I also calculated FLOPs and parameters myself.

But what is strange is that the parameter I calculated is 6.71M instead of 6.73M in the paper. At the same time, FLOPs are the same as the results in the paper.

Attach my code:

    from ptflops import get_model_complexity_info

    with torch.cuda.device(0):
        model = SalsaNext(nclasses=20)
        macs, params = get_model_complexity_info(model, (5, 64, 2048), as_strings=True,
                                                 print_per_layer_stat=True, verbose=True)
        print('{:<30}  {:<8}'.format('Computational complexity: ', macs))
        print('{:<30}  {:<8}'.format('Number of parameters: ', params))

    # Computational complexity:       62.84 GMac   1 Mac = 2 FLOPs
    # Number of parameters:           6.71 M

Do you have any suggestions for reproducing the results in the paper?

Best,
Iris

I use 8 TitanX Pascal GPUs to train the model following the README, and get the error below.

RuntimeError: CUDA out of memory. Tried to allocate 384.00 MiB (GPU 0; 11.91 GiB total capacity; 10.95 GiB already allocated; 250.56 MiB free; 11.15 GiB reserved in total by PyTorch)

Can you supply the hardware requirements? @TiagoCortinhal

Hi,

I am able to train the salsanext model with your training infrastructure and the Kitti Dataset. Now, we are trying to train the Salsanext with our own data and our own infrastructure (pytorch lightning). First we had to adapt it, because our frame format is different, and after 4 poolings we reached odd numbers which prevents further pooling. So we went for the easier path, removing some layers. We also are not using the preprocessing in a first approach.
Now, the data size is matching with the layers, but we have the issue that loss is always NAN. As our data propagate into the layers, the more and more NANs are appearing in the layer results. Of course, we have some initial nans in our lidar frame (all the points which don t have any reflection), but not more than in the Kitti dataset. We found out, that if we replace all the LeakyRelu by normal nn.Relu, this is not the case, and we get a converging training loss. Do you have any idea?

We have some further tests to do to get a clearer idea. For instance, we are using the Adam loss function and not the Crossentropy, and we could also change our learning rate, but the fact that the LeakyRelu is working for Kitti data and not for our data is intriguing. So we thought that maybe you also had similar issues while implementing your model.

In your demo you show images mapped to point cloud prediction. Where you have downloaded them?

Thank you for opening the source code of this briliant work!

The values of Number of Parameters and FLOPs are shown in the paper.

I'm curious how do you calculate these two values?
Which tools/packages do you use for calculation?

Best,
Iris

@TiagoCortinhal can you share the pre-trained model ??

Hello,

• My aim to run inference without retraining
• I am running on Intel CPU (i.e. no GPU)
• I am using the pre-trained model provided in this repo.
• I got the complete datasets from SemanticKITTI.
• I setup everything as per the instruction provided.

When I run eval script with all variables, I get error in user.py on line 226 (which I believe is doing a forward pass on the model).
The error says:

"Illegal instruction (core dumped)"

Because of this, the Evaluation script fails when checking for length of pred_names (off course the inference never finished correctly)
I concluded this with some print statements in user.py and infer.py files.

Any help or debug tips you can provide? Thank You in advance!

Terminal Output :

(salanext) ~/zebra/NNs/SalsaNext/SalsaNext-master$ ./eval.sh -d dataset -p logs_preds -m saved_model -s train -c 30

INTERFACE:
dataset /home/mipso/zebra/NNs/SalsaNext/SalsaNext-master/dataset
log /home/mipso/zebra/NNs/SalsaNext/SalsaNext-master/logs_preds
model /home/mipso/zebra/NNs/SalsaNext/SalsaNext-master/saved_model
Uncertainty False
Monte Carlo Sampling 30
infering train

---

Opening arch config file from /home/mipso/zebra/NNs/SalsaNext/SalsaNext-master/saved_model
Opening data config file from /home/mipso/zebra/NNs/SalsaNext/SalsaNext-master/saved_model
train 00
train 01
train 02
train 03
train 04
train 05
train 06
train 07
train 09
train 10
valid 08
test 11
test 12
test 13
test 14
test 15
test 16
test 17
test 18
test 19
test 20
test 21
model folder exists! Using model from /home/mipso/zebra/NNs/SalsaNext/SalsaNext-master/saved_model
Sequences folder exists! Using sequences from /home/mipso/zebra/NNs/SalsaNext/SalsaNext-master/dataset/sequences
parsing seq 00
parsing seq 01
parsing seq 02
parsing seq 03
parsing seq 04
parsing seq 05
parsing seq 06
parsing seq 07
parsing seq 09
parsing seq 10
Using 19130 scans from sequences [0, 1, 2, 3, 4, 5, 6, 7, 9, 10]
Sequences folder exists! Using sequences from /home/mipso/zebra/NNs/SalsaNext/SalsaNext-master/dataset/sequences
parsing seq 08
Using 4071 scans from sequences [8]
Sequences folder exists! Using sequences from /home/mipso/zebra/NNs/SalsaNext/SalsaNext-master/dataset/sequences
parsing seq 11
parsing seq 12
parsing seq 13
parsing seq 14
parsing seq 15
parsing seq 16
parsing seq 17
parsing seq 18
parsing seq 19
parsing seq 20
parsing seq 21
Using 20351 scans from sequences [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]

---

Cleaning point-clouds with kNN post-processing
kNN parameters:
knn: 5
search: 5
sigma: 1.0
cutoff: 1.0
nclasses: 20

---

Infering in device: cpu
*** JS INFO: Entering infer ****
*** JS INFO: Entering infer_subset ****
*** JS INFO: self.gpu = False
*** JS INFO: Entering infer_subset NO uncertainty ****

Illegal instruction (core dumped)

finishing infering.
Starting evaluating

---

INTERFACE:
Data: /home/mipso/zebra/NNs/SalsaNext/SalsaNext-master/dataset
Predictions: /home/mipso/zebra/NNs/SalsaNext/SalsaNext-master/logs_preds
Split: train
Config: config/labels/semantic-kitti.yaml
Limit: None

---

Opening data config file config/labels/semantic-kitti.yaml
Ignoring xentropy class 0 in IoU evaluation
[IOU EVAL] IGNORE: tensor([0])
[IOU EVAL] INCLUDE: tensor([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
19])
Traceback (most recent call last):
File "./evaluate_iou.py", line 237, in
eval(DATA["split"][FLAGS.split],splits,FLAGS.predictions)
File "./evaluate_iou.py", line 67, in eval
len(label_names) == len(pred_names))
AssertionError

I want to infer semantic-kitti dataset with pre-trained model.
When I run eval.sh, it required data_cfg.yaml. What information should I provide to run this code successfully?
BTW, I use salsanext.yml as arch_cfg.yaml, is that correct?

Thank you!

Thanks for your great work.
I had a question in inference code from user.py.
The order of model seems to be (mean, var)
But in user.py:
https://github.com/Halmstad-University/SalsaNext/blob/71114947a019c4760d52a4717cac531de12c67a2/train/tasks/semantic/modules/user.py#L157
And the proj_output2 is used to get classification result:
https://github.com/Halmstad-University/SalsaNext/blob/71114947a019c4760d52a4717cac531de12c67a2/train/tasks/semantic/modules/user.py#L159

I am looking forward to your reply!

Hi, could you please kindly tell me the example gif in your readme belong to which sequence in the semantic Kitti? Or the order of the sequence index in your video is what?

Hi,

I tried infering labels using the pretrained model, and it worked great without uncertainty

Network seq 00 scan 000000.label in 0.7649648189544678 sec
KNN Infered seq 00 scan 000000.label in 0.0006988048553466797 sec
Network seq 00 scan 000001.label in 0.049358367919921875 sec
KNN Infered seq 00 scan 000001.label in 0.0001277923583984375 sec
Network seq 00 scan 000002.label in 0.03960108757019043 sec
KNN Infered seq 00 scan 000002.label in 0.00012493133544921875 sec
Network seq 00 scan 000003.label in 0.03216409683227539 sec
KNN Infered seq 00 scan 000003.label in 0.00012087821960449219 sec
Network seq 00 scan 000004.label in 0.03187704086303711 sec
KNN Infered seq 00 scan 000004.label in 0.00012159347534179688 sec
Network seq 00 scan 000005.label in 0.03268098831176758 sec
KNN Infered seq 00 scan 000005.label in 0.0001220703125 sec
Network seq 00 scan 000006.label in 0.035898447036743164 sec
KNN Infered seq 00 scan 000006.label in 0.0001232624053955078 sec
Network seq 00 scan 000007.label in 0.03408312797546387 sec
KNN Infered seq 00 scan 000007.label in 0.0001232624053955078 sec
Network seq 00 scan 000008.label in 0.032814741134643555 sec
KNN Infered seq 00 scan 000008.label in 0.00012636184692382812 sec
Network seq 00 scan 000009.label in 0.0343012809753418 sec
KNN Infered seq 00 scan 000009.label in 0.0001857280731201172 sec

Using -u (and requiring a little fix as mentionned here : #12 (comment)) it seems to work and generates log_var and uncert label files but is much slower, (more than 6 seconds per sequence, thus 200 times slower for 30 iterations).

Infered seq 00 scan 000000.label in 7.025984764099121 sec
7.025984764099121
Infered seq 00 scan 000001.label in 6.477065801620483 sec
6.751525282859802
Infered seq 00 scan 000002.label in 6.456561803817749 sec
6.653204123179118
Infered seq 00 scan 000003.label in 6.463520765304565 sec
6.60578328371048
Infered seq 00 scan 000004.label in 6.522738695144653 sec
6.589174365997314
Infered seq 00 scan 000005.label in 6.484813451766968 sec
6.571780880292256
Infered seq 00 scan 000006.label in 6.5031116008758545 sec
6.56197098323277
Infered seq 00 scan 000007.label in 6.512105464935303 sec
6.555737793445587
Infered seq 00 scan 000008.label in 6.457853555679321 sec
6.544861767027113
Infered seq 00 scan 000009.label in 6.514480829238892 sec
6.541823673248291
Infered seq 00 scan 000010.label in 6.4799792766571045 sec
6.536201455376365

I also reduced the number of iterations with -mc 10 instead of the default 30, but it still takes around 3 seconds per sequence inference.

Is there a particular reason for such a difference ?

@TiagoCortinhal ,
I train the SalsaNext and in the log folder three checkpoints are saved, SalsaNext, SalsaNext_train_best and SalsaNext_valid_best.
In SalsaNext_valid_best, the best_val_iou and valid_iou are 0.6101. However, I test SalsaNext_valid_best using infer.py and evaluate_iou.py , the average iou is 0.590, which is different from 0.6101.
Why are they different?

Hello!!
I'm sorry to ask you a lot of questions during your busy schedule.
I have some questions about how to project LiDAR segmentation results onto a camera image.

1. Is it correct that the program that projects the LiDAR segmentation results onto RGB images uses visualization.py ?

2. If you are using visualization.py, how do you actually project them onto RGB images?
   It seems that it can be done by adding basedir, sequence, uncerts, and so on.
   And then, I think that it is possible to run it with $ python visualization.py, but it cannot be done well.

basedir = '~/SalsaNext/pred-valid'
sequence = '08'
uncerts = ' '
preds = 'predictions'
gt = ' '
img = '~/Dataset/KITTI/dataset/sequences/08/image_2/'
lidar = '~/Dataset/SemanticKITTI/dataset/sequences/08/'
projected_uncert = ' '
projected_preds = ' '

Each value is tentatively defined as above. and these blanks, I could not understand which path to specify.
As a result of executing it for the time being, it became as follows.

$ python3 visualization.py
Ground truth poses are not avaialble for sequence 08.
Traceback (most recent call last):
  File "visualization.py", line 78, in <module>
    color_map_dict = yaml.safe_load(open("color_map.yml"))['color_map']
FileNotFoundError: [Errno 2] No such file or directory: 'color_map.yml'

• What are uncerts, gt, projented_uncert, and projected_preds ?
• What is color_map.yml ?
• Is it possible to project to RGB with Seq-08 ?

Thank you for your cooperation.

1. Is your SalsaNext result the same as SalsaNextUncertainty? I downloaded your pretrained model and did inference on kitti validation set sequence 08. SalsaNextUncertainty looks worse than SalsaNext at frame 0.
   SalsaNext
   
   SalsaNextUncertainty
   

2. What are your data and model uncertainty numbers?
   For kitti validation set sequence 08:
   In log_var folder file 000000.label, the first 10 numbers are [-0.01334856, 0.00081099, -0.02349238, -0.02350008, -0.02083466,-0.0205034 , -0.02259915, -0.01983254, -0.01969955, -0.00620079]
   In uncert folder file 000000.label, the first 10 numbers are [0.2471824 , 0.21661215, 0.38824964, 0.36705604, 0.38694522, 0.3721131 , 0.35623956, 0.33999357, 0.33527 , 0.3765529 ]

Are those numbers correct? I would expect the log_var number is lower because you set the data variance to 2e-7 and uncert also lower because in the paper you showed the model variance is roughly 1e-4.

3. SalsaNextUncertainty are outputing weird results on some frames. For example kitti validation set sequence 08 frame 21:
   
   The predictions are all 0. The uncert and log_var are all nan.

Thank you!

The previous issues could have been found with continuous integration. Adding GitHub Workflows to run linting and maybe tests if available could help here.

Hello,
I'm a French student and I make an internship onto semantic segmentation, so I need to run your code, but the link to the pretrained model seems dead. If you have the time, can you update your link or send me another ?
Thank you !
(Sorry if it's not the place to ask that I'm not very familiar with Github)

Hi! According to sentences on top left of Page.2， a mistake may lie in Block Ⅰ of Fig.2, the kernel size might be 3,5,7 and the dilation rates of three dilated convolution layers should keep the same.

https://github.com/Halmstad-University/SalsaNext/blob/cc8c75dc68d2607d16e2c82be61e7254e5b74a12/train/tasks/semantic/modules/user.py#L152

Should we set dropout to training when we do monte-carlo while inference?
Just like deep_uncertainty_estimation repo did:
https://github.com/uzh-rpg/deep_uncertainty_estimation/blob/0093e12f234ad20da5a97aebb57ba060c8c4ca75/eval.py#L177
Thank you.

Hi,
I have read both SalsaNet and SalsaNext, and they are great work.
I noticed that you have change the skip connection in docoder from add to concat. Have you compared the performance between them, and what is the qualitative results.

Thanks.

I am very new to this field. I have 3d lidar point cloud data and videos. My task is to do semantic segmentation of that data for further investigation.
Can anyone help me that how can I do that. Can I use any pre-traind model. And how.

Nice to meet you.
I am a student studying semantic segmentation in Japan.

I'm interested in SalsaNext announced by your team and trying to implement it.
However, I am having trouble with the behavior related to prediction.

Specifically, when I run the "Eval Script" posted by your team using the pretrained model, I got the following error message:

labels:  4071
predictions:  0
Traceback (most recent call last):
  File "./evaluate_iou.py", line 237, in <module>
    eval(DATA["split"][FLAGS.split],splits,FLAGS.predictions)
  File "./evaluate_iou.py", line 66, in eval
    assert (len(label_names) == len(scan_names) and
AssertionError

When I checked, "label_names" and "scan_names" were entered normally, but the cause was that "len ​​(pred_name) = 0".
This is because there is no prediction directory, and I prepared an empty directory for the time being.

So, could you please teach me how to eliminate this error?
If there is a "Test Script" etc. before the "Eval Script", would you please post it if possible?
Thank you.

Could you give me some advices on how to train the two hard sem.seg classes: Motorcyclist and Other-ground.In my experiment, They are completely close to 0.
Notice that in the pre-trained model,there also nearly 0 , as follow.

train_update : 0.00010502038594856752
train_loss : 0.35978143828302084
train_acc : 0.9487481341277788
train_iou : 0.7022054133173021
valid_loss : 1.4024676781763712
valid_acc : 0.9072582666167708
valid_iou : 0.6136875937614279
best_train_iou : 0.7011966601875725
best_val_iou : 0.6177246216099593
train_hetero : 0
unlabeled : 0.,
car : 0.9437
bicycle :0.4768
motorcycle : 0.5139
truck : 0.8114
other-vehicle : 0.4819
person : 0.7155
bicyclist : 0.8241
motorcyclist : 0.
road : 0.9459
parking : 0.3692
sidewalk : 0.8166
other-ground : 0.0103
building : 0.8843
fence : 0.5629
vegetation : 0.8482
trunk : 0.6664
terrain : 0.7131
pole : 0.6047
traffic-sign : 0.4711
valid_heteros : 0
But in your paper,the results are much better.(Motorcyclist:19.4 Other-ground:29.1)
Could you share tyour experience?

Hi!

From your ResBlock class, I can see that you use a constant downsampling rate of 2 by using the nn.AvgPool2d layer with kernel_size=3, stride=2 and padding=1.

However, in your arXiv paper, the first residual block downsamples the width from 2048 to 512 which indicates a downsampling rate of 4. Also, I don't understand how the last layer upsamples the feature map from 1024x64x32 to 2048x64x32 since in your code, a Conv2d layer with kernel_size=(1,1) is used here.

Is this a mistake in the visualization of the architecture?

Thank you!

The folder structure was changed, but not the code.
SalsaNextUncertainty was renamed into SalsaNextAdf

This is my infer information

Opening arch config file from /home/arl/SalsaNext/TrainEvalResources/SalsaNextTrainedModel [Errno 20] Not a directory: '/home/arl/SalsaNext/TrainEvalResources/SalsaNextTrainedModel/arch_cfg.yaml' Error opening arch yaml file.
I downloaded the train model, named it "SalsaNextTrainedModel" and put it in "/home/arl/SalsaNext/TrainEvalResources/"

I do not understand why it is looking for arch_cfg.yaml there. Also arch_cfg.yaml does not exist anywhere in the repo that I cloned.

2. I have the following, not sure this is correct:
   ./eval.sh
-d /home/arl/SalsaNext/TrainEvalResources/Eval-dataset # location=folder of images to test
-p /home/arl/SalsaNext/TrainEvalResources/SaveLabelPredictions # folder of where to put prediction results
-m /home/arl/SalsaNext/TrainEvalResources/SalsaNextTrainedModel # folder/name of downloaded trained model
-s valid # eval on validation set, using ```-s validation``` as per instructions does not work
-n salsanextExp1 # name of experiment
-c 0 # required to avoid error 3. below

3. ./infer.py: error: argument --monte-carlo/-c: invalid int value: ''
   Fixed by setting -c 0 in arg list

4. ModuleNotFoundError: No module named 'tasks.semantic.modules.SalsaNextUncertainty'
   Fixed by changing SalsaNextUncertainty to SalsaNextAdf at line 20 of user.py

Hey, I just wanted to know if there is a reason for the extra class in the pretrained model?

user.py In current master has:

from tasks.semantic.modules.SalsaNextUncertainty import *

I would suggest importing the new module name instead:

from tasks.semantic.modules.SalsaNextAdf import *

infer.py tries to load arch_cfg.yaml and data_cfg.yaml. Neither file can be found.

    # open arch config file
    try:
        print("Opening arch config file from %s" % FLAGS.model)
        ARCH = yaml.safe_load(open(FLAGS.model + "/arch_cfg.yaml", 'r'))
    except Exception as e:
        print(e)
        print("Error opening arch yaml file.")
        quit()

    # open data config file
    try:
        print("Opening data config file from %s" % FLAGS.model)
        DATA = yaml.safe_load(open(FLAGS.model + "/data_cfg.yaml", 'r'))
    except Exception as e:
        print(e)
        print("Error opening data yaml file.")
        quit()

Hi,
Thanks for your generous opening source of this briliant project!

I have two questions for the project:

1. Does pretrained model use uncertainty or not during training? Is this pretrained model the one which could reproduce 59.5 point-wise mean-IoU in the Table I of paper?

2. When I train the model on my self, I found that there is a serious overfitting problem.
   The pictures below are my training loss and valid loss curve.
   I want to ask why this problem occurs?
   Do you also have this problem during training?

I am looking forward to your reply!