Loss extremely high and doesn't decrease about casenet HOT 5 CLOSED

Hi,

One thing you could try is increasing the radius on this line here to 4 or even more (to thicken edges in ground truths). This could be done in the case that ground truth edges aren't prominent enough.

A few things you could try are

1. Use a warmup training strategy (train a network for a few ~5 epochs, and use it as pre-trained network)
2. Increase batch size if you can, or utilize gradient accumulation.
3. Play around with learning rate, or follow the paper if you wish (as per the paper, "we set the learning rate, step size, gamma and crop size to 1e − 7 / 5e − 8, 10000 / 20000, 0.1 / 0.2 and 352×352 / 472×472 respectively for SBD and Cityscapes").
4. The most important point, you'll need to train for a large number of epochs. I had trained for ~200 and saw some semblance of a decent results. The authors train for 40000 epochs, and I think that this may be the issue in your case.

Hope that I was of some help, and that you can solve the problem.

from casenet.

Hi,

Yes, while I didn't run training on cityscapes, I trained it on a custom dataset. The loss I got was extremely high too, in the best case it was ~20000. I didn't create an evaluation script, so I can't say for sure, but the visualizations gave me a hint whether it was acceptable or not.

Apologies if this wasn't very helpful!

from casenet.

Thank you for your reply.
In my case, the loss didn't decrease at all and the results were not that good. Could you please give a hint which parameters could I tune to improve the output?

from casenet.

@anirudh-chakravarthy hi，it's 40000 iteration，not 40000 epoch。

from casenet.

Hi,

One thing you could try is increasing the radius on this line here to 4 or even more (to thicken edges in ground truths). This could be done in the case that ground truth edges aren't prominent enough.

A few things you could try are

1. Use a warmup training strategy (train a network for a few ~5 epochs, and use it as pre-trained network)
2. Increase batch size if you can, or utilize gradient accumulation.
3. Play around with learning rate, or follow the paper if you wish (as per the paper, "we set the learning rate, step size, gamma and crop size to 1e − 7 / 5e − 8, 10000 / 20000, 0.1 / 0.2 and 352×352 / 472×472 respectively for SBD and Cityscapes").
4. The most important point, you'll need to train for a large number of epochs. I had trained for ~200 and saw some semblance of a decent results. The authors train for 40000 epochs, and I think that this may be the issue in your case.

Hope that I was of some help, and that you can solve the problem.

Hello! I trained on my own dataset, the loss was xe+07 but after one epoch it decreased to -1e+06. How could this situation happen? Why did the loss decreased to negative value?

from casenet.