Having trouble reproducing exact numbers for evaluation. about dinov2 HOT 6 OPEN

I have to revise my issue from above a bit.
I found a little issue in my evaluation that messed up my results.
The numbers I get now are closer to the ones reported in the paper, but not exactly the same:

Val: 86.55 (81.146) -> in paper: 86.5 (81.1)
Real: 89.705 (86.649) -> in paper: 89.6 (86.6)
v2: 78.41 (71.14) -> in paper 78.4 (70.9)
Inet-C: 27.286 (53.422) -> in paper 28.2 (54.4)
Inet-A: 76.08 (34.373) -> in paper 75.9 (33.5)
Inet-R: 79.253 (55.08) -> in paper 78.8 (53.7)
Inet-Sketch: 62.794 (42.166)  -> in paper 62.5 (41.2) 
Inet-Style: 34.578 (13.51) -> no evaluation in paper

I also verified my evaluation with other models. I was able to reproduce the results reported here for the ResNet50 (for ImageNet-Real I don't have a number to compare to).
I don't think it would be beneficial if I share my entire code here, but I'll try to put together a minimal example once I have the time.

from dinov2.

Sure. I am not an author of the paper. So it is fine. I was just trying to see if there was still an open issue.

from dinov2.

Would you be able to share your full code for this evaluation ?

from dinov2.

The results I get for ViT-g/14 (ViT-S/14) are:

Val: 86.55 (81.146) -> in paper: 86.5 (81.1)
Real: 89.705 (86.649) -> in paper: 89.6 (86.6)
v2: 78.41 (71.14) -> in paper 78.4 (70.9)

The results I get for ViT-g/14 (ViT-S/14) are:

Inet-A: 76.08 (34.373) -> in paper 75.9 (33.5)
Inet-R: 79.253 (55.08) -> in paper 78.8 (53.7)
Inet-C: 27.286 (53.422) -> in paper 28.2 (54.4)
Inet-Sketch: 62.794 (42.166)  -> in paper 62.5 (41.2) 

So it is fine, the issue can be closed, right?
You get the same results (or better), except for Inet-C, but you are close.

from dinov2.

I'm still wondering why I can't reproduce the exact values you report in the paper (especially the OOD evaluation).
These are the parameters that were used to generate the tables in the paper, right?
Deviations of this magnitude are a bit weird, so I'm trying to understand what's going on.

from dinov2.

These are the parameters that were used to generate the tables in the paper, right?

Not always, typically the average of multiple evaluations using different seeds for training is reported.
Also maybe is a random process in augmenting data for evaluation. double check.

from dinov2.