May be a wrong code ? about pfenet HOT 7 CLOSED

请问，在validate中有两层for循环，一层循环10次，一层遍历val_loader， 请问外面的那10次循环目的是什么呢？ 另外我怕我英语说不清楚，我再说下我上面的问题，就是我输出subcls发现它的形状是(5, 20), 可是这样subcls[0].cpu().numpy()[0] 这样获取的是第一个query-support集合的类别，可是一个mini_batch中包含20个query-support集合啊，不同的query-support的类别不一定是一样的啊，问什么只取subcls[0].cpu().numpy()[0]

from pfenet.

作者您看我自己的理解对不对，我感觉外层的for循环应该紧贴在我截图的代码外面，subcls[0].cpu().numpy()[0]也应改成subcls[i].cpu().numpy()[i]，而对于外面的for循环是10，是因为默认的batch_size_val 是10

from pfenet.

@zhijiejia

Because our code evaluates the prediction by comparing it with the original label without resizing operation and different ground-truth labels are in different sizes, the batch size for validation can only be set as one, which is the reason why we only adopt "subcls[0].cpu().numpy()[0]".

If you'd like to evaluate prediction with the altered labels (for example resizing all labels to a fixed size), you can accordingly modify the code in your way by supporting evaluation with multiple batches. While this may cause unfair comparisons with the previous method because they may use different sizes for altering labels (like 417). Normally, in semantic segmentation, the size of the ground truth label should not be altered.

from pfenet.

Thanks, I remember wrong batch_ size_ val value, but why is there a for loop in the outer layer of validate?

from pfenet.

@zhijiejia

This loop is used for additional evaluation steps.

PASCAL only has 1400+ images for validation and for some splits there are only 300-400 images containing the corresponding categories. However, evaluation with 300-400 steps causes unstable results since the support samples are randomly selected for each query image.

Previous practices evaluate with 1,000 steps, but the results are still unstable in different runs. Therefore, in our paper, we instead recommend evaluating few-shot segmentation methods with 5,000 steps on PASCAL, which is approx. 10 times of the images of each split (each split has been evaluated with approx. 10 different support samples) and this helps minimize the performance variation.

So in our code, you can see that we stop the evaluation when to total step number is larger than the pre-defined test_num, or 10 epochs have been evaluated.

from pfenet.

Thanks for your careful answer, I get it.

from pfenet.

@zhijiejia

If you have any further questions, feel free to drop me a message :)

from pfenet.