nltk-maxent-pos-tagger

nltk-maxent-pos-tagger is a part-of-speech (POS) tagger based on Maximum Entropy (ME) principles written for NLTK. It is based on NLTK's Maximum Entropy classifier (nltk.classify.maxent.MaxentClassifier), which uses MEGAM for number crunching.

Part-of-Speech Tagging

nltk-maxent-pos-tagger uses the set of features proposed by Ratnaparki (1996), which are also used in his MXPOST implementation (Java).

Installation

1. Install Python and NLTK.

NLTK offers lots of data sets, which you might download and install from within a Python shell:

import nltk
nltk.download()

Download at least brown or treebank, as nltk-maxent-pos-tagger uses them for its demo() function.

Usage

Have a look at the example given in the demo() function in mxpost.py. Basically, you just have to import the tagger and train it with labelled data to use it:

import mxpost
maxent_tagger = mxpost.MaxentPosTagger()
maxent_tagger.train(tagged_training_sentences)

for sentence in unlabeled_sentences:
    maxent_tagger.tag(sentence)

Meta

Status: Beta. I wrote this in 2008 as a semester project for a class on NLP tools.
Licence: GPL Version 3
Author: Arne Neumann [email protected]

TODO

1. speed / memory consumption
   As you can expect, a Python implementation is much slower and consumes much more RAM than similar tools written in Java or C/C++ (MXPOST, acopost, C&C etc.). This being said, most of the time isn't spend in Python but rather in MEGAM (which is written in O'Caml and therefore shouldn't have such issues). NLTK currently is only able to encode POS features explicitly when converting data for MEGAM. According to the MEGAM website, using implicit feature encoding should be much faster.

2. accuracy
   I trained several taggers on the WSJ corpus (90% training / 10% test data). nltk-maxent-pos-tagger achieved an accuracy of 93.64% (100 iterations, rare feature cutoff = 5) while MXPOST reached 96.93% (100 iterations). Since both implementations use the same feature set, results shouldn't be that different. Unfortunately, there's no source code available for MXPOST, but comparing nltk-maxent-pos-tagger with OpenNLP's implementation should be helpful.