Comments (1)
Good question. This is not properly documented I guess.
You have to enable the right settings for the SemEval annotation scheme. With the --semeval
option this is done automatically (see the semeval()
function). If you run on a single file as in your example, you need to specify the post-processing options explicitly:
$ python3 coref.py --fmt=semeval2010 --exclude=relpronouns,reflexives,reciprocals,predicatives,appositives,npsingletons,relpronounsplit data/semeval2010NLdevparses/file_7 >/tmp/file_7.conll
$ python3 ~/code/coval/scorer.py data/manualeval/knack_file7.gold.conll /tmp/file_7.conll
recall precision F1
mentions 75.00 78.95 76.92
muc 70.00 70.00 70.00
bcub 65.00 72.81 68.68
ceafe 70.09 77.88 73.78
ceafm 75.00 78.95 76.92
lea 60.00 68.42 63.93
CoNLL score: 70.82
This is with current master, and this score is slightly lower than the one in the paper. Some of the improvements that I have made after the paper was published are apparently regressions for this particular document. Let's reproduce the score from the paper:
$ git checkout 5b33ab4e2f4bc29619d3f3fdbef462ccd4807aef
[...]
$ python3 coref.py --fmt=semeval2010 --exclude=relpronouns,reflexives,reciprocals,predicatives,appositives,npsingletons,relpronounsplit data/semeval2010NLdevparses/file_7 >/tmp/file_7.conll
Traceback (most recent call last):
[...]
ValueError: unrecognized --exclude argument: reflexives
$ python3 coref.py --fmt=semeval2010 --exclude=relpronouns,reflectives,reciprocals,predicatives,appositives,npsingletons data/semeval2010NLdevparses/file_7 >/tmp/file_7.conll
$ python3 ~/code/coval/scorer.py data/manualeval/knack_file7.gold.conll /tmp/file_7.conll
recall precision F1
mentions 70.00 87.50 77.78
muc 70.00 77.78 73.68
bcub 60.00 80.21 68.65
ceafe 60.09 85.84 70.70
ceafm 70.00 87.50 77.78
lea 55.00 75.00 63.46
CoNLL score: 71.01
Notice how the post processing options are slightly different, but the score is reproduced exactly. Let me know if you run into anything else!
from dutchcoref.
Related Issues (3)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dutchcoref.