Ontonotes train/test/split about dilated-cnn-ner HOT 9 CLOSED

Can you be more specific about which results you tried to replicate?

I used the data here: http://conll.cemantix.org/2012/data.html which is train-v4, dev-v4 and test-v9.

from dilated-cnn-ner.

Data we both used are same,
I tried to replicate the results from your paper Fast and Accurate Entity Recognition with Iterated Dilated Convolutions
On the test I got F1 score 64 , but in the paper F1 score is around 86

from dilated-cnn-ner.

from dilated-cnn-ner.

from dilated-cnn-ner.

I tested the dilated-cnn config ...... it seems correct to me.... not that far away :)

Best dev F1: 84.55

Segment evaluation (test):
	        F1	Prec	Recall
Micro (Seg)	85.37	85.60	85.14
Macro (Seg)	72.14	73.70	70.65
-------
   ORDINAL	80.31	80.10	80.51
       LOC	69.92	67.89	72.07
   PRODUCT	52.70	54.17	51.32
      NORP	92.38	91.78	92.98
WORK_OF_ART	48.52	47.67	49.40
  LANGUAGE	44.44	57.14	36.36
     MONEY	83.68	83.28	84.08
   PERCENT	88.32	87.82	88.83
    PERSON	90.85	91.57	90.14
       ORG	83.37	83.79	82.95
  CARDINAL	82.71	83.85	81.60
       GPE	93.79	94.60	92.99
      TIME	58.39	60.30	56.60
      DATE	83.21	81.58	84.89
       FAC	62.17	62.88	61.48
       LAW	61.76	75.00	52.50
     EVENT	45.87	54.35	39.68
  QUANTITY	70.97	68.75	73.33
Processed 152728 tokens with 11257 phrases; found: 11196 phrases; correct: 9584.
Testing time: 44 seconds

from dilated-cnn-ner.

@strubell One last question F1 score presented in the Fast and Accurate Entity Recognition with Iterated Dilated Convolutions paper is F1_macro or F1_micro ?

from dilated-cnn-ner.

Micro

from dilated-cnn-ner.

Thanks!

from dilated-cnn-ner.

Hello, I have a question about the datasplit. From your earlier comment.

I used the data here: http://conll.cemantix.org/2012/data.html which is train-v4, dev-v4 and test-v9.

In test-v9 as far as I can tell there are 11,057 entities while in the test-v4 section there are 11,257 like you mention in Table 8. Which test set did you use? From the look of the preprocess.sh script it lookes like it was the v4 because the names of the files are gold_conll rather than gold_parse_conll like in the v9 test set.

I used this to find entites find annotations/ -name '*.v4_gold_conll' | grep -v 'pt/nt' | xargs cat | sed 's/\s\s*/ /g' | cut -d' ' -f11 | sed -n '/^(/p' | wc -l

Thanks for your clarification.

from dilated-cnn-ner.