Comments (4)
Sure, here is the code. It assumes that there is a train.txt, dev.txt and test.txt in the folder with IOB encoding. It creates then train.txt.bio ...
"""
Converts the IOB encoding from CoNLL 2003 to BIO encoding
"""
filenames = ['train.txt', 'dev.txt', 'test.txt']
for filename in filenames:
fOut = open(filename+'.bio', 'w')
fIn = open(filename, 'r')
for line in fIn:
if line.startswith('-DOCSTART-'):
lastChunk = 'O'
lastNER = 'O'
continue
if len(line.strip()) == 0:
lastChunk = 'O'
lastNER = 'O'
fOut.write("\n")
continue
splits = line.strip().split()
chunk = splits[2]
ner = splits[3]
if chunk[0] == 'I':
if chunk[1:] != lastChunk[1:]:
chunk = 'B'+chunk[1:]
if ner[0] == 'I':
if ner[1:] != lastNER[1:]:
ner = 'B'+ner[1:]
splits[2] = chunk
splits[3] = ner
fOut.write("\t".join(splits))
fOut.write("\n")
lastChunk = chunk
lastNER = ner
from emnlp2017-bilstm-cnn-crf.
No, I didn't use the POS information. I would also recommend not to use it, because at inference you would need a POS tagger to detect the named entities in a sentence. Further, adding POS information does not improve the performance of the classifier.
Training on CoNLL 2003 NER is rather straight forward. Due to copyright issues I sadly cannot share the dataset, but here are the steps:
- Convert the strange IOB encoding in the original CoNLL 2003 dataset to an BIO encoding. See issue #22 why this is needed. If you need a script to convert from IOB to BIO, let me know.
- The files for CoNLL 2003 NER contains lines that start with -DOCSTART- => remove these lines, they are meta data from the dataset to indicate that a new document starts.
- Training is similar to the Train_Chunking.py . Only change the dataset description:
datasets = {
'conll2003_ner':
{'columns': {0:'tokens', 3:'NER_BIO'},
'label': 'NER_BIO',
'evaluate': True,
'commentSymbol': None}
}
from emnlp2017-bilstm-cnn-crf.
Thanks.
from emnlp2017-bilstm-cnn-crf.
Can you please share the script for converting IOB to BIO encoding?
from emnlp2017-bilstm-cnn-crf.
Related Issues (20)
- Where to modify the random seed value? HOT 2
- Learning rate HOT 4
- Multi-task POS/NER parallel data HOT 7
- CoNLL.py Getting an Error in Train_Chunking.py HOT 2
- Multi-task settings HOT 6
- about attention mechanism... HOT 1
- Use a pre-trained model for evaluating a new test set? HOT 4
- Transfer Learning using Save_and_Load.py HOT 2
- Accuracy HOT 5
- Error while computing F1-score HOT 7
- TypeError: add_weight() got multiple values for argument 'name' HOT 5
- Killed HOT 1
- Question HOT 4
- Question HOT 4
- Adding new embeddings to a trained model. HOT 4
- I want to do NER about English instead of German,How can i modify HOT 1
- Is it entity level evaluation or else token level evaluation used when computing f1 scores? HOT 1
- Adding new word features (prefix/suffix) for training
- AttributeError: 'Node' object has no attribute 'input_masks'
- as_list() is not defined on an unknown TensorShape.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emnlp2017-bilstm-cnn-crf.