How to handle completely wrong sentence word? about symspell HOT 1 OPEN

This is a common problem. It is not only that the user intentionally types something wrong, but there are always unknown words in the input text that are not in the dictionary.

So how we can distinguish a non-existing or unknown word from a misspelled word?

Solution1: restrict the maximum edit distance (the number of splits + the numer of spelling corrections) within a sliding text window. If a string needs to be split into too many small words, and almost all of the sub-words need an additional spelling correction we can assume that this is an unknown/non-existing word.

Solution2: Use n-gram probabilities or Markov-chains. The co-occurrence of words is not random. Some words are more likely to occur together in a sentence than others, some words are frequently follow each others, others never. if the n-gram probabilities of the split and corrected words are below a certain threshold, we can assume that this is not a genuine correction, but an unknown/non-existing word.

Both solutions are currently not part of SymSpell and need to be implemented as an extension or by modifying the SymSpell code.

from symspell.