The fairness of different implementations of MSE about cufar HOT 6 CLOSED

Thanks for your attention and consideration. All experimental results (including CUFAR and baselines) we reported in the paper are evaluated in the same MSE approach, so it is fair enough for comparison.

We also evaluate all models using two different implementations of MSE, and we get the same results from these two implementations.

from cufar.

This seems to be the precision issue for numerical calculations. In most cases, these two implementations have the same MSE error, but sometimes they have a certain deviation. Their final results are different, when the deviations of all samples are accumulated. Please see the picture below

Another question： why not directly use the official results of the comparison methods?
`

from cufar.

I do not think that such little and small deviations (e-6 or even smaller in your pic) may cause a huge difference, i.e., 0.3 MSE in the pic. Moreover, I just evaluated TaskBJ-P1, results can be found in the attached file P1_test_results.txt. It shows that there is no deviation between two implementations of MSE.

For a fair comparison, we implemented all baselines and our model in the same experimental environment.

from cufar.

The version of dependencies is as follows:
scikit-learn 1.0.2
numpy 1.21.5

from cufar.

I find the ture reason.

I call the np.mean(np.power(real - pred, 2)) function after get_MAPE(pred, real).
However, the value of real has been modified in get_MAPE(pred, real)，since its arguments are reference variables.

from cufar.

Congrats! Please contact me if you have any problem.

from cufar.