Wrong precision and recall in entity linking evaluation about hipe-scorer HOT 5 CLOSED

Hello, and thanks for your message.

This is not really a bug but rather a limitation of the annotation scheme. As the entity linking part of the README says

"as there is no IOB-tagging, a consecutive row of identical links is considered as a single entity.”

Without IOB tags on link annotations it is not possible to determine when a new entity/link starts. If the same entity link (e.g. NIL) is predicted consecutively, the scorer cannot know whether it spans across one or more entities. This behaviour should affect consecutive NIL links exclusively since systems assign links on the basis of mentions and it would be very unlikely to have twice the same QID for two consecutive (and different) mentions. Even if, two consecutive and same non-NIL QIDs should not have an impact, but we will check how many consecutive NIL links span across more than one entity in the reference.

from hipe-scorer.

Hello again,

I've been thinking on how this could be solved. And I think the main reason of the previous issues, is that the predictions are not being split into spans using the gold standard spans, either defined by NER tags or consecutive QID.

For the following gold standard:

1 Neuchâld    B-loc    O    B-loc.adm.reg    O    O    O    Q69345    _    LED0.33
2 SI    B-pers    O    B-pers.ind    O    B-comp.title    O    Q78068340    _    NoSpaceAfter|LED0.09
3 .    I-pers    O    I-pers.ind    O    I-comp.title    O    Q78068340    _    LED0.09
4 la    O    O    O    O    O    O    _    _    _
5 Confédération    B-loc    O    B-loc.adm.nat    O    O    O    Q39    _    LED0.00
6 Suisse    I-loc    O    I-loc.adm.nat    O    O    O    Q39    _    NoSpaceAfter|LED0.00
7 est O    O    O    O    O    O    _    _    _

You can define that gold standard spans as: [1,1] -> Q69345; [2,3] -> Q78068340; [4, 4] -> _; [5,6] -> Q39; [7,7] -> _. Yes, I am including the _ as a span of size 1 but they would be removed after. This is done to check that the prediction didn't include a QID in a place where it shouldn't.

Then for the predictions:

1 Neuchâld    B-loc    O    B-loc.adm.reg    O    O    O    NIL    _
2 SI    B-pers    O    B-pers.ind    O    B-comp.title    O    NIL    _
3 .    I-pers    O    I-pers.ind    O    I-comp.title    O    NIL    _
4 la    O    O    O    O    O    O    Q39|Q340787|Q568452    _
5 Confédération    B-loc    O    B-loc.adm.nat    O    O    O    Q39|Q340787|Q568452    _
6 Suisse    I-loc    O    I-loc.adm.nat    O    O    O    Q39|Q340787|Q568452    _
7 est    O    O    O    O    O    O   _    _

You can split using the gold standard spans to get the predicted QIDs: [1,1] -> NIL; [2,3] -> NIL; [4, 4] -> Q39; [5,6] -> Q39; [7,7] -> _ . Then, based on the spans generate the values for the evaluation and removing the "_" cases, you can have the following data:

Gold standard: Q69345, Q78068340, Q39
Predictions: NIL, NIL, Q39, Q39

In the case one span in the predictions contain more than one QID, you could subsplit them using consecutive QIDs like in:

1 Neuchâld    B-loc    O    B-loc.adm.reg    O    O    O    NIL    _
2 SI    B-pers    O    B-pers.ind    O    B-comp.title    O    NIL    _
3 .    I-pers    O    I-pers.ind    O    I-comp.title    O    NIL    _
4 la    O    O    O    O    O    O    Q39|Q340787|Q568452    _
5 Confédération    B-loc    O    B-loc.adm.nat    O    O    O    Q39|Q340787|Q568452    _
6 Suisse    I-loc    O    I-loc.adm.nat    O    O    O    Q38    _
7 est    O    O    O    O    O    O   _    _

Then have: [1,1] -> NIL; [2,3] -> NIL; [4, 4] -> Q39; [5,6] -> Q39,Q38; [7,7] -> _ and in consequence:

Gold standard: Q69345, Q78068340, Q39
Predictions: NIL, NIL, Q39, Q39, Q38

This first error explained in the original post (i.e. consecutive predicted NILs) would be fixed using gold standard spans defined either by consecutive QID or boundaries from NER. The second error (consecutive NILs in the gold standard) would only by fixed if the spans come from the NER boundaries.

In all the cases, I think this evaluation would be more balanced, specially when multiple NILs are predicted.

from hipe-scorer.

Hello,

I've seen that you're preparing for HIPE 2022. I would just like to tell you that I applied the modifications some time ago to a fork of the scorer that you can find here:

https://github.com/creat89/CLEF-HIPE-2020-scorer

You can test it and see if the changes would be good to be used in HIPE 2022.

from hipe-scorer.

Great, thanks a lot. We are resolving this currently.

from hipe-scorer.

this was fixed by the PR, thanks again @creat89 !

from hipe-scorer.