Comments (4)
Hi @SefaZeng
This issue also happens with my code: in-valid transitions (e.g. O I-PER) are produced by the BiLSTM-CRF model.
The issue is sadly not trivial and I don't know how to fix it.
The CRF is initialized with random probabilities for the transitions, i.e. O I-PER can be as likely as O B-PER. Of course, the CRF does not know anything from the encoding and about allowed transitions.
During training, these transition probabilities are updated, so that the CRF learns that O I-PER is unlikely. However, it converges rather slowly to a 0 probability. This makes sense, as how should the CRF be able to distinguish that O I-PER is not possible at all and 'it is rare but I haven't seen enough data'.
With more epochs, the number of invalid tags usually converge to a low number or even to zero in my experiments.
As I solution what I use is a post-processing step: The code checks whether the tags from the CRF are valid BIO-encoded. If it finds an invalid tag, it sets this tag to O.
from emnlp2017-bilstm-cnn-crf.
Hi @SefaZeng
This issue also happens with my code: in-valid transitions (e.g. O I-PER) are produced by the BiLSTM-CRF model.The issue is sadly not trivial and I don't know how to fix it.
The CRF is initialized with random probabilities for the transitions, i.e. O I-PER can be as likely as O B-PER. Of course, the CRF does not know anything from the encoding and about allowed transitions.
During training, these transition probabilities are updated, so that the CRF learns that O I-PER is unlikely. However, it converges rather slowly to a 0 probability. This makes sense, as how should the CRF be able to distinguish that O I-PER is not possible at all and 'it is rare but I haven't seen enough data'.
With more epochs, the number of invalid tags usually converge to a low number or even to zero in my experiments.
As I solution what I use is a post-processing step: The code checks whether the tags from the CRF are valid BIO-encoded. If it finds an invalid tag, it sets this tag to O.
Can I set the initial states to zero to avoid this problem?
from emnlp2017-bilstm-cnn-crf.
@SefaZeng I think that could work, however, you would need to ensure to get the mapping right. Especially when the number of tags changes (e.g. you add B-LOC and I-LOC to your tagset), you must ensure that you set the zeros at the right place. Otherwise it can easily happen that B-LOC => I-LOC is initialized with a zero probability.
Further, the CRF is bi-directional, i.e. not only the previous label is important but also the next label determines which label is produced. This can make it rather complicated to initialize the CRF correctly.
from emnlp2017-bilstm-cnn-crf.
@nreimers Emmm.. I set the initializer of U, b_start, b_end and initial state in the viterbi_decode to zeros,but it doesn't work.Maybe post-processing is the only way.
But I am still confusing why it will happen.Because in statistic opinion, if the in-valid transitions never appear in the data,the probability or maybe the weights in the neural network should be very low or only zero.
from emnlp2017-bilstm-cnn-crf.
Related Issues (20)
- Where to modify the random seed value? HOT 2
- Learning rate HOT 4
- Multi-task POS/NER parallel data HOT 7
- CoNLL.py Getting an Error in Train_Chunking.py HOT 2
- Multi-task settings HOT 6
- about attention mechanism... HOT 1
- Use a pre-trained model for evaluating a new test set? HOT 4
- Transfer Learning using Save_and_Load.py HOT 2
- Accuracy HOT 5
- Error while computing F1-score HOT 7
- TypeError: add_weight() got multiple values for argument 'name' HOT 5
- Killed HOT 1
- Question HOT 4
- Question HOT 4
- Adding new embeddings to a trained model. HOT 4
- I want to do NER about English instead of German,How can i modify HOT 1
- Is it entity level evaluation or else token level evaluation used when computing f1 scores? HOT 1
- Adding new word features (prefix/suffix) for training
- AttributeError: 'Node' object has no attribute 'input_masks'
- as_list() is not defined on an unknown TensorShape.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emnlp2017-bilstm-cnn-crf.