Pre-trained model Performance inconsistent with the one in README about self-attentive-parser HOT 3 CLOSED

The "benepar_en" model is just a compressed version of the model we report in our paper. Sorry if that was unclear. We follow the standard splits: 2-21 for training, 22 for dev, and 23 for test. If you'd like to check any of the details, the code is all in the repo and the command to train the model is in the README (search for --use-elmo).

What evaluation are you using that's getting almost 96 F1? I really wish the parser was that accurate, but that isn't the number I've gotten in any of my evaluations. The 95.07 number was produced by evalb (see the EVALB directory in the repo) using the COLLINS.prm parameter file, and includes sentences of all lengths (not just up to length 40). I know that when I was getting started with parsing it took me a while to track down what the standard evaluation procedure is, especially since most parsing papers aren't very detailed on this point. Is it possible that you're running the evaluation differently?

from self-attentive-parser.

Hi,

Thanks for the reply! I am also new to parsing. plus I just want to use the output of your model for my own task. That means the model I have been using have only seen the data of wsj 02-21 sections, right?

What's more, I just would like to check a few things with you in case I have made some silly mistakes:

1. your wsj corpus comes from https://catalog.ldc.upenn.edu/ldc99t42
2. did you use any other scripts to process the wsj parsing tree before training? It seems there is some differences (in terms of formating) with your 22 section output and my WSJ parsing tree.
3. the 95.07 actually comes from the "Bracketing FMeasure" right?

Thanks for your time and help!

from self-attentive-parser.

Hi,

OK. I finally found my mistake which happened in constructing the golden parsing tree from WSJ corpus. Thanks for your help!! I will close the issue.

from self-attentive-parser.