I am realizing that there is an issue with reading large wordlists that have a lot of junk in them that is formatted poorly. (Not valid UTF-8)
I will probably just change the way I am reading files. This will probably be fixed fairly easily.
I discovered a bug in WordlistGenerator that can cause it to crash unexpectedly when reading some Markov Chain files. I am working on a fix. I don't think this is a bug to worry about if you are using wordlists that only use ASCII characters. Anyways, a fix should be out soon.
I've tried the tools on some different small wordlists, but when I run sort | uniq afterwards it's shrinked by something like 80%, suggesting a rather large output of duplicates.