Code Monkey home page Code Monkey logo

Comments (1)

andreasvc avatar andreasvc commented on July 22, 2024

Good question. This is not properly documented I guess.

You have to enable the right settings for the SemEval annotation scheme. With the --semeval option this is done automatically (see the semeval() function). If you run on a single file as in your example, you need to specify the post-processing options explicitly:

$ python3 coref.py --fmt=semeval2010 --exclude=relpronouns,reflexives,reciprocals,predicatives,appositives,npsingletons,relpronounsplit data/semeval2010NLdevparses/file_7 >/tmp/file_7.conll
$ python3 ~/code/coval/scorer.py data/manualeval/knack_file7.gold.conll /tmp/file_7.conll
             recall  precision         F1
mentions      75.00      78.95      76.92
muc           70.00      70.00      70.00
bcub          65.00      72.81      68.68
ceafe         70.09      77.88      73.78
ceafm         75.00      78.95      76.92
lea           60.00      68.42      63.93
CoNLL score:  70.82

This is with current master, and this score is slightly lower than the one in the paper. Some of the improvements that I have made after the paper was published are apparently regressions for this particular document. Let's reproduce the score from the paper:

$ git checkout 5b33ab4e2f4bc29619d3f3fdbef462ccd4807aef
[...]
$ python3 coref.py --fmt=semeval2010 --exclude=relpronouns,reflexives,reciprocals,predicatives,appositives,npsingletons,relpronounsplit data/semeval2010NLdevparses/file_7 >/tmp/file_7.conll
Traceback (most recent call last):
[...]
ValueError: unrecognized --exclude argument: reflexives
$ python3 coref.py --fmt=semeval2010 --exclude=relpronouns,reflectives,reciprocals,predicatives,appositives,npsingletons data/semeval2010NLdevparses/file_7 >/tmp/file_7.conll
$ python3 ~/code/coval/scorer.py data/manualeval/knack_file7.gold.conll /tmp/file_7.conll
             recall  precision         F1
mentions      70.00      87.50      77.78
muc           70.00      77.78      73.68
bcub          60.00      80.21      68.65
ceafe         60.09      85.84      70.70
ceafm         70.00      87.50      77.78
lea           55.00      75.00      63.46
CoNLL score:  71.01

Notice how the post processing options are slightly different, but the score is reproduced exactly. Let me know if you run into anything else!

from dutchcoref.

Related Issues (3)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.