Comments (5)
Hello, and thanks for your message.
This is not really a bug but rather a limitation of the annotation scheme. As the entity linking part of the README says
"as there is no IOB-tagging, a consecutive row of identical links is considered as a single entity.”
Without IOB tags on link annotations it is not possible to determine when a new entity/link starts. If the same entity link (e.g. NIL) is predicted consecutively, the scorer cannot know whether it spans across one or more entities. This behaviour should affect consecutive NIL links exclusively since systems assign links on the basis of mentions and it would be very unlikely to have twice the same QID for two consecutive (and different) mentions. Even if, two consecutive and same non-NIL QIDs should not have an impact, but we will check how many consecutive NIL links span across more than one entity in the reference.
from hipe-scorer.
Hello again,
I've been thinking on how this could be solved. And I think the main reason of the previous issues, is that the predictions are not being split into spans using the gold standard spans, either defined by NER tags or consecutive QID.
For the following gold standard:
1 Neuchâld B-loc O B-loc.adm.reg O O O Q69345 _ LED0.33
2 SI B-pers O B-pers.ind O B-comp.title O Q78068340 _ NoSpaceAfter|LED0.09
3 . I-pers O I-pers.ind O I-comp.title O Q78068340 _ LED0.09
4 la O O O O O O _ _ _
5 Confédération B-loc O B-loc.adm.nat O O O Q39 _ LED0.00
6 Suisse I-loc O I-loc.adm.nat O O O Q39 _ NoSpaceAfter|LED0.00
7 est O O O O O O _ _ _
You can define that gold standard spans as: [1,1] -> Q69345; [2,3] -> Q78068340; [4, 4] -> _; [5,6] -> Q39; [7,7] -> _. Yes, I am including the _ as a span of size 1 but they would be removed after. This is done to check that the prediction didn't include a QID in a place where it shouldn't.
Then for the predictions:
1 Neuchâld B-loc O B-loc.adm.reg O O O NIL _
2 SI B-pers O B-pers.ind O B-comp.title O NIL _
3 . I-pers O I-pers.ind O I-comp.title O NIL _
4 la O O O O O O Q39|Q340787|Q568452 _
5 Confédération B-loc O B-loc.adm.nat O O O Q39|Q340787|Q568452 _
6 Suisse I-loc O I-loc.adm.nat O O O Q39|Q340787|Q568452 _
7 est O O O O O O _ _
You can split using the gold standard spans to get the predicted QIDs: [1,1] -> NIL; [2,3] -> NIL; [4, 4] -> Q39; [5,6] -> Q39; [7,7] -> _ . Then, based on the spans generate the values for the evaluation and removing the "_" cases, you can have the following data:
Gold standard: Q69345, Q78068340, Q39
Predictions: NIL, NIL, Q39, Q39
In the case one span in the predictions contain more than one QID, you could subsplit them using consecutive QIDs like in:
1 Neuchâld B-loc O B-loc.adm.reg O O O NIL _
2 SI B-pers O B-pers.ind O B-comp.title O NIL _
3 . I-pers O I-pers.ind O I-comp.title O NIL _
4 la O O O O O O Q39|Q340787|Q568452 _
5 Confédération B-loc O B-loc.adm.nat O O O Q39|Q340787|Q568452 _
6 Suisse I-loc O I-loc.adm.nat O O O Q38 _
7 est O O O O O O _ _
Then have: [1,1] -> NIL; [2,3] -> NIL; [4, 4] -> Q39; [5,6] -> Q39,Q38; [7,7] -> _ and in consequence:
Gold standard: Q69345, Q78068340, Q39
Predictions: NIL, NIL, Q39, Q39, Q38
This first error explained in the original post (i.e. consecutive predicted NILs) would be fixed using gold standard spans defined either by consecutive QID or boundaries from NER. The second error (consecutive NILs in the gold standard) would only by fixed if the spans come from the NER boundaries.
In all the cases, I think this evaluation would be more balanced, specially when multiple NILs are predicted.
from hipe-scorer.
Hello,
I've seen that you're preparing for HIPE 2022. I would just like to tell you that I applied the modifications some time ago to a fork of the scorer that you can find here:
https://github.com/creat89/CLEF-HIPE-2020-scorer
You can test it and see if the changes would be good to be used in HIPE 2022.
from hipe-scorer.
Great, thanks a lot. We are resolving this currently.
from hipe-scorer.
this was fixed by the PR, thanks again @creat89 !
from hipe-scorer.
Related Issues (18)
- ToDo CLEF scorer HOT 11
- This issue was a mistake
- Fuzzy results are named `'ent_type'` in json output
- [Feature request] : Make the scorer directly accessible via a python API
- Extend tagset.txt to include entity tags from all HIPE-2022 datasets HOT 1
- Mention the truncation danger somewhere
- Evaluation Measures: Understanding of macro average
- Make file urls amenable to the scorer HOT 2
- Add "pip install" support HOT 3
- CSV reader bug: "quotechar" must be a 1-character string
- Evaluation measures: Slot error rates HOT 3
- weighting of partial matches in the fuzzy evaluation scoring HOT 4
- enforce submission requirements HOT 1
- last small things before making the repo public HOT 1
- evaluation NEL with multiple links HOT 1
- There are no tags in the system response for the column... HOT 3
- handling of invalid tags vs out-of-GT tags
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hipe-scorer.