How to convert semantic segmentation results into required format about semantic-aware-scene-recognition HOT 5 OPEN

Because 0 might be a semantic label. Which in the case of the ADE20K dataset its related to "Wall" class.

from semantic-aware-scene-recognition.

Hi!

Thanks for your question!

If your semantic segmentation model outputs a score tensor Y with size L x W x H (with normalized probabilities, i.e. between 0 and 1) you need to sort Y and select the 3 labels with a higher probability for each pixel. By that, you will have the Top@3 labels with their respective Top@3 scores.

You can do that on the fly in the training and evaluation loop, however, in order to save time, we decided to save that labels and scores to images so the prediction is only done once. To save 3 semantic labels per pixel we decided to encode them into the three channels of an RGB image.

from semantic-aware-scene-recognition.

Thank for your explanation. I still have one further question. According to you paper, the semantic segmentation score is set to "0", if it is not one of the top3 predictions. However, there are many zero regions in the segmentation-precomputed SUN397 data. How is that possible?

Best,
Neo

from semantic-aware-scene-recognition.

For example, sem_score file './Data/Datasets/SUN397/noisy_scores_RGB/val/airplane_cabin/sun_akzqlgepekqslhbn.png' contains many zero values. I don't think there should be zero values.

from semantic-aware-scene-recognition.

if i understand correctly, the original output from semantic segmentation network is a Scores (150xWxH) matrix.

1. we normalize the distribution at each pixel ==> torch.sum(Score [i,:,:]) == 1. the output of this stage is Scores_norm (150xWxH)
2. we generate the sem_scores (3xWxH) by extracting the top3 scores at each pixel. sem_scores[0,i,j] represents the top-1 scores at pixel(i,j).
3. we retrieve the index of top3 scores as sem_labels (3xWxH). The matrix is divided by 255.

Following such procedure, the value of sem_score cannot be zero. Do i miss any steps?

from semantic-aware-scene-recognition.