Comments (3)
Hironsan
commented on July 30, 2024
1
You just forgot to specify mode and scheme to classification_report. If it's specified correctly, the result is the same:
def evaluate(y_true, y_pred, scheme, average):
print(precision_score(y_true, y_pred, average=average, mode='strict', scheme=scheme), end='\t')
print(recall_score(y_true, y_pred, average=average, mode='strict', scheme=scheme), end='\t')
print(f1_score(y_true, y_pred, average=average, mode='strict', scheme=scheme))
print(classification_report(y_true, y_pred, digits=3, mode='strict', scheme=scheme))
# IOBES
0.6666666666666666 0.8 0.7272727272727272
precision recall f1-score support
LOC 0.667 0.800 0.727 10
micro avg 0.667 0.800 0.727 10
macro avg 0.667 0.800 0.727 10
weighted avg 0.667 0.800 0.727 10
# BILOU
0.6666666666666666 0.8 0.7272727272727272
precision recall f1-score support
LOC 0.667 0.800 0.727 10
micro avg 0.667 0.800 0.727 10
macro avg 0.667 0.800 0.727 10
weighted avg 0.667 0.800 0.727 10from seqeval.
Hironsan
commented on July 30, 2024
Please show me the evaluation snippet and the data.
from seqeval.
rsuwaileh
commented on July 30, 2024
I generated a small example from my dataset:
z_true = [['O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'E-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'I-LOC', 'E-LOC', 'O', 'S-LOC', 'O', 'S-LOC', 'O', 'O', 'O', 'O', 'O', 'O'],
['O', 'O', 'O', 'O', 'O', 'O', 'S-LOC', 'S-LOC', 'O', 'O', 'O', 'O', 'O'],
['O', 'O', 'B-LOC', 'I-LOC', 'E-LOC', 'O', 'S-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O'],
['O', 'B-LOC', 'E-LOC', 'O', 'O', 'B-LOC', 'E-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]
z_pred = [['O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'E-LOC', 'B-LOC', 'I-LOC', 'E-LOC', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'I-LOC', 'E-LOC', 'O', 'S-LOC', 'O', 'S-LOC', 'O', 'O', 'O', 'O', 'O', 'O'],
['O', 'O', 'O', 'O', 'O', 'O', 'S-LOC', 'S-LOC', 'B-LOC', 'I-LOC', 'E-LOC', 'O', 'O'],
['O', 'S-LOC', 'O', 'O', 'O', 'O', 'S-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O'],
['O', 'O', 'O', 'B-LOC', 'E-LOC', 'B-LOC', 'E-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]
scheme = IOBES
average = "micro"
evaluate(z_true, z_pred, scheme, average)
The results I get:
0.6666666666666666 0.8 0.7272727272727272
precision recall f1-score support
LOC 0.667 0.800 0.727 10
micro avg 0.667 0.800 0.727 10
macro avg 0.667 0.800 0.727 10
weighted avg 0.667 0.800 0.727 10
When I change the scheme to BILOU using the same example and lables above:
z_true = [['O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'L-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'I-LOC', 'L-LOC', 'O', 'U-LOC', 'O', 'U-LOC', 'O', 'O', 'O', 'O', 'O', 'O'],
['O', 'O', 'O', 'O', 'O', 'O', 'U-LOC', 'U-LOC', 'O', 'O', 'O', 'O', 'O'],
['O', 'O', 'B-LOC', 'I-LOC', 'L-LOC', 'O', 'U-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O'],
['O', 'B-LOC', 'L-LOC', 'O', 'O', 'B-LOC', 'L-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]
z_pred = [['O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'L-LOC', 'B-LOC', 'I-LOC', 'L-LOC', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'I-LOC', 'L-LOC', 'O', 'U-LOC', 'O', 'U-LOC', 'O', 'O', 'O', 'O', 'O', 'O'],
['O', 'O', 'O', 'O', 'O', 'O', 'U-LOC', 'U-LOC', 'B-LOC', 'I-LOC', 'L-LOC', 'O', 'O'],
['O', 'U-LOC', 'O', 'O', 'O', 'O', 'U-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O'],
['O', 'O', 'O', 'B-LOC', 'L-LOC', 'B-LOC', 'L-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]
scheme = BILOU
average = "micro"
evaluate(z_true, z_pred, scheme, average)
I get the same P, R, & F1. However, the report is different. I'm using micro average with both schemes:
0.6666666666666666 0.8 0.7272727272727272
precision recall f1-score support
LOC 0.625 0.556 0.588 9
micro avg 0.625 0.556 0.588 9
macro avg 0.625 0.556 0.588 9
weighted avg 0.625 0.556 0.588 9
This is the evaluate function that uses seqeval:
def evaluate(y_true, y_pred, scheme, average):
print(precision_score(y_true, y_pred, average = average, mode='strict', scheme=scheme), end='\t')
print(recall_score(y_true, y_pred, average = average, mode='strict', scheme=scheme), end='\t')
print(f1_score(y_true, y_pred, average = average, mode='strict', scheme=scheme))
print(classification_report(y_true, y_pred, digits=3))
from seqeval.
Related Issues (20)
- Understanding NER F1 score seqeval calculation
- SyntaxError: future feature annotations is not defined
- error: subprocess-exited-with-error HOT 1
- Installation on GitHub `windows-latest` randomly fails HOT 2
- Use of seqeval locally with Hugging face evaluate HOT 1
- Documentation?
- Proposal to get TP, FP, FN of all targets
- hope add a new method to caculate metric: remove the tag header(B-, I-) and only used tag body(Person, Loc) with entity boundries to caculate metric.
- Accuracy Vs F1-score HOT 1
- digits doesn't work when set `output_dict=True`
- Req. is Missing?
- get_entities
- classification report throws 'int' object has no attribute 'startswith' HOT 1
- How to disable seqeval label formatting for POS-tagging HOT 2
- Consideration of <PAD> token while evaluating NER tasks HOT 4
- F1 score for missing class prediction
- is seqeval.metrics.f1_score compatible with aspect extraction tasks
- No module named 'seqeval' even after installing the package HOT 2
- Does it support BILO format? Also I noticed strange result with BILOU format
- O-labels treatment HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from seqeval.