Include Moses Tokenizer about text HOT 9 CLOSED

I know NLTK has a Moses tokenizer, but unsure if it's a good port / I've always used tokenize.pl anyway

For the reference, for large corpora (e.g MT) I've found it faster to pretokenize your data and feed it into TorchText with the str.split tokenizer, versus having to tokenize on every run.

from text.

@nelson-liu Is there an option to include a bash script to tokenize your data as part of the library?

This issue mentions that Moses in NLTK was fixed nltk/nltk#1214

from text.

Hmm, not directly. Ostensibly you could write a function that takes an input string and runs the bash script on it (e.g with subprocess) and parse the output for tokenization, but that feels like a lot of overhead

from text.

Edited my comment above. Looks like NLTK contributors fixed Moses in this merged pull request: nltk/nltk#1553

from text.

I think including the NLTK version of the Moses tokenizer is a good idea, though it shouldn’t be difficult to use the existing API to call it for now

from text.

+1, slighty unrelated but wonder if it's worth including the other NLTK tokenizers (e.g punkt or PTB) / having some sort of public facing API for using the spacy tokenizers in other languages. This seems like a slippery slope API design-wise, though...

from text.

Yeah, I think I’d rather show in the docs etc how easy it is to call them yourself. But lots of people (including me until a few minutes ago!) don’t know that NLTK now has a Moses-compatible tokenizer, so it might be worth including that one so more people know they can move away from the perl script preprocessing approach.

from text.

from text.

PR at #58

from text.