Label distributions about nlvr HOT 2 CLOSED

Regarding the distributions:

1. Yes, this is due to removing examples with low agreement.
2. The validation process we ran was able to completely re-label image/sentence pairs as true or false. We didn't consider the original position of the image when labeling it. In practice, 8.2% of examples had a different label than the one inferred by the original position. This number (2176/3696) could also be confounded by the fact that not all of those sentences have 4 examples. I'm not sure what the distribution of labels is if you look at sentences with four images (of the 3358 you identified).
3. This is also caused by the re-labeling during validation, but also may be due to the number of images for those 496 sentences. It's possible that many of those 496 only have one or two associated images, which makes it more likely they will all have the same label.

Also, I believe the identifier tells you the original position of the image in the four. If n = 0 or 1, then the original position was at the top (sentence should have been true); if n = 2 or 3, then the original position was at the bottom (sentence should have been false).

Hope this helps.

from nlvr.

That's helpful. Thanks!

from nlvr.