Wrong transition in crf when doing a sequence labeling task about emnlp2017-bilstm-cnn-crf HOT 4 OPEN

Hi @SefaZeng
This issue also happens with my code: in-valid transitions (e.g. O I-PER) are produced by the BiLSTM-CRF model.

The issue is sadly not trivial and I don't know how to fix it.

The CRF is initialized with random probabilities for the transitions, i.e. O I-PER can be as likely as O B-PER. Of course, the CRF does not know anything from the encoding and about allowed transitions.

During training, these transition probabilities are updated, so that the CRF learns that O I-PER is unlikely. However, it converges rather slowly to a 0 probability. This makes sense, as how should the CRF be able to distinguish that O I-PER is not possible at all and 'it is rare but I haven't seen enough data'.

With more epochs, the number of invalid tags usually converge to a low number or even to zero in my experiments.

As I solution what I use is a post-processing step: The code checks whether the tags from the CRF are valid BIO-encoded. If it finds an invalid tag, it sets this tag to O.

from emnlp2017-bilstm-cnn-crf.

Hi @SefaZeng
This issue also happens with my code: in-valid transitions (e.g. O I-PER) are produced by the BiLSTM-CRF model.

The issue is sadly not trivial and I don't know how to fix it.

The CRF is initialized with random probabilities for the transitions, i.e. O I-PER can be as likely as O B-PER. Of course, the CRF does not know anything from the encoding and about allowed transitions.

During training, these transition probabilities are updated, so that the CRF learns that O I-PER is unlikely. However, it converges rather slowly to a 0 probability. This makes sense, as how should the CRF be able to distinguish that O I-PER is not possible at all and 'it is rare but I haven't seen enough data'.

With more epochs, the number of invalid tags usually converge to a low number or even to zero in my experiments.

As I solution what I use is a post-processing step: The code checks whether the tags from the CRF are valid BIO-encoded. If it finds an invalid tag, it sets this tag to O.

Can I set the initial states to zero to avoid this problem?

from emnlp2017-bilstm-cnn-crf.

@SefaZeng I think that could work, however, you would need to ensure to get the mapping right. Especially when the number of tags changes (e.g. you add B-LOC and I-LOC to your tagset), you must ensure that you set the zeros at the right place. Otherwise it can easily happen that B-LOC => I-LOC is initialized with a zero probability.

Further, the CRF is bi-directional, i.e. not only the previous label is important but also the next label determines which label is produced. This can make it rather complicated to initialize the CRF correctly.

from emnlp2017-bilstm-cnn-crf.

@nreimers Emmm.. I set the initializer of U, b_start, b_end and initial state in the viterbi_decode to zeros,but it doesn't work.Maybe post-processing is the only way.
But I am still confusing why it will happen.Because in statistic opinion, if the in-valid transitions never appear in the data,the probability or maybe the weights in the neural network should be very low or only zero.

from emnlp2017-bilstm-cnn-crf.