rriegs / ngram Goto Github PK

As a first step in this project, it is necessary to collect a large catalog of word senses. The most obvious source for these is Wiktionary, given that data dumps of the entire Wiktionary corpus are readily available online and that Wiktionary's definitions have extremely regular formatting.

It is of interest to extract headwords, parts of speech, and definitions for each sense of each dictionary entry. I am inclined to do this in perl because of its excellent regex functionality and XML parsing utilities.