Comments (3)
Interesting: The current scorer still relies on many preprocessing steps that are idiosyncratically bound to the HIPE format and evaluation scenario. In a way it could still be seen as an evaluation space. https://huggingface.co/evaluate-metric (similar to GLUE).
from hipe-scorer.
@simon-clematide (hoping I did not misunderstood) I donβt see it as such a problem that the metric depends on the annotation style (domain-dependent). I was raising the issue mainly because it could be easier to integrate on my side for the training of different models and of course, it could be easier to integrate in a metric such as seqeval.
from hipe-scorer.
One can see the metric as instead of multitask (CoNLL with columns for each task eg NER, chunking), some type of multilevel (columns in HIPE) - multilevel-seqeval π
from hipe-scorer.
Related Issues (18)
- ToDo CLEF scorer HOT 11
- Wrong precision and recall in entity linking evaluation HOT 5
- This issue was a mistake
- Fuzzy results are named `'ent_type'` in json output
- [Feature request] : Make the scorer directly accessible via a python API
- Extend tagset.txt to include entity tags from all HIPE-2022 datasets HOT 1
- Mention the truncation danger somewhere
- Evaluation Measures: Understanding of macro average
- Make file urls amenable to the scorer HOT 2
- CSV reader bug: "quotechar" must be a 1-character string
- Evaluation measures: Slot error rates HOT 3
- weighting of partial matches in the fuzzy evaluation scoring HOT 4
- enforce submission requirements HOT 1
- last small things before making the repo public HOT 1
- evaluation NEL with multiple links HOT 1
- There are no tags in the system response for the column... HOT 3
- handling of invalid tags vs out-of-GT tags
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hipe-scorer.