Your predictions error is inconsistent with your paper, why? about fcrn-depthprediction HOT 2 OPEN

Hi, thanks for pointing this one out!

We did indeed get the errors that we have reported in the paper. This is what we see in our machines, using Matlab version 2017a or older:

However, we looked more into this matter and came across the following problems:

• All operations were performed in single precision (as both the network output and the depth ground truth came in single precision). However, as we sum and divide over all pixels, it turns out that their number is way too large to be handled by single precision and this leads to precision errors. We would like to report that correct evaluation requires double precision and thus we will update all reported errors (rel: 0.140, rmse: 0.597). This does not affect the delta measures.

• This understanding came when we tested the predictions with Matlab 2017b. With this version, it looks like all operations are internally performed with double precision, even when the matrices are provided as single. Therefore, using our code in 2017b one can get accurate results.

• In general, we observed that different versions of Matlab might cause different results, especially on different machines. It depends on different factors, such as 32-bit/64-bit, the C compiler used, etc. Similar issues have been reported: e.g. https://uk.mathworks.com/matlabcentral/answers/438-why-are-computational-results-in-matlab-sometimes-different-on-different-machines-and-how-can-i-prev

• There might also be small differences in the output between python/matlab scripts, because of the above.

• While investigating this, we also realized that there are perhaps small differences in how different methods evaluate their predicted depth maps. We believe that most methods use a certain center crop (as suggested by the NYU Depth dataset) in which the depth values are valid. This generally reduces errors (rel: 0.136, rmse: 0.584 in double precision). For completeness and fairness of comparisons, we will now report errors using this evaluation protocol as well.

• We will of course update accordingly the error metrics calculation script as soon as possible.

from fcrn-depthprediction.

So to make sure I understood this correctly:

1. The results in the paper are inaccurate and you're going to publish an update to the paper to include the correct results.

Or,

2. The results in the paper are correct, and the evaluation code used by the op is wrong.

Can someone confirm?

from fcrn-depthprediction.