Add IOB encoding issue about emnlp2017-bilstm-cnn-crf HOT 2 CLOSED

Hi @fuleying
The CoNLL 2003 datatset for NER comes in a quite strange encoding, which i named IOB (not to be confused with BIO encoding).

The B-tag (Begin-tag) is only used if two different named entities with the same tag are follow directly each other, for example (thats how it is encoded in the dataset):
I love London => O O I-LOC
I love London Berlin and Paris => O O I-LOC B-LOC O I-LOC
I love London Harry Potter => O O I-LOC I-PER I-PER

In my paper I evaluated BIO encoding (as used in CoNLL 2000 chunking), IOB encoding (as used in CoNLL 2003 NER) and IOBES encoding.

IOB encoding perfoms quite badly, and to be honest, it is a totally strange format. BIO and IOBES encoding perform on par.

So when working with the CoNLL 2003 NER dataset, it recommend to first convert it from IOB encoding to BIO encoding before training a classifier.

I hope this answers your question.

from emnlp2017-bilstm-cnn-crf.

Hi @nreimers Thanks for your answers.

from emnlp2017-bilstm-cnn-crf.