bot breaking up words with single quotation marks in them about twitchmarkovchain HOT 3 CLOSED

Hello!

Since cf14dda I indeed split up punctuation from other tokens. The goal here was to make e.g. hello, into two tokens, rather than just one, so the bot learns how to extend from hello. Similarly, from I've it learns I followed by 've, so that it can extend I.

The issue you're seeing here is that it actually does not know how to extend ["I", "'ve"], because of a bug that converted I've into I 've when generating, but kept the word as I've when learning. I've fixed that now.

However, another small issue still exists: My "detokenizer", i.e. the function to turn ca n't back into can't, isn't always perfect:

>>> TreebankWordDetokenizer().tokenize(["Do", "n't"])
"Do n't"
>>> TreebankWordDetokenizer().tokenize(["Do", "n't", "want"])
"Don't want"
>>> TreebankWordDetokenizer().tokenize(["I", "'ve"])  
"I 've"
>>> TreebankWordDetokenizer().tokenize(["I", "'ve", "seen"])
"I've seen"

Using NLTK's TreebankWordDetokenizer. I can't really explain this at the moment, but it requires a fix from NLTK.

Thanks for reporting this!

• Tom Aarsen

from twitchmarkovchain.

This should be fixed now in the latest release. Feel free to upgrade. Your old database won't immediately update, but new entries should be correct now.

from twitchmarkovchain.

I've opened up an issue on NLTK for the tokenizing issue I mentioned: nltk/nltk#3069.

from twitchmarkovchain.