mlaves / well-calibrated-regression-uncertainty Goto Github PK

Thank you for your work, it has been very helpful to me, but I have a few questions I need to ask you:

1. Why is dropout set to true during the validation process in the train.py file? I think that dropout should only be enabled during the inference process for obtaining uncertainty with MC dropout.
2. Does sigma calibration help with OOD (Out-Of-Distribution) detection? If uncertainties of all samples are multiplied by a fixed positive number, the relative size of the uncertainty values for OOD and ID (In-Distribution) samples will not change. Moreover, when I read the code, I found that the images drawn for OOD are also based on the results before sigma calibration, without involving sigma calibration
3. Recently, when conducting OOD detection on my dataset, I found that the overall uncertainty values for OOD samples are lower compared to in-domain samples, what do you think could be the reason?
   Thank you again for your answers

Hi Max, I am a physics student trying to use a Bayesian neural network to fit a dataset. I am not an expert in the field of deep learning so forgive me if this question sounds stupid.
I was trying to use your technique but i don't have much data and I was thinking of using the same training set to get the scaling factor instead of a separate calibration set. I read in Levi's article that the latter option is suggested but he uses a non-Bayesian probabilistic network with determined weights and perhaps feared overfitting. Since you have used it with a Bayesian network, I wanted to ask you if you think it is really necessary to use a separate set or is it possible to use the same training set. Thank you and sorry for the trouble and for the wall-of-text.