wolfgarbe / symspell Goto Github PK

is there any paper for symspell?
dose symspell detect real word error?

I'm trying to use SymSpell for OCR post processing spell correction.
I have noticed that, SymSpell LookupCompound excluding Numbers and Special characters from the output. In my context, numbers and characters are really important for further analysis.
Is it possible to avoid Numbers and Special characters elimination?

Version: SymSpell 6.3 C# project

Steps to reproduce:

1. Build the SymSpell C# code

2. Go to \SymSpell\SymSpell.CompoundDemo

3. Run dotnet run .

4. Enter below input
   "To find out more about how we use information, visit or contact-any of our offices 24/7"

5. It gives below output.
   to find out more about how we use information visit or contact any of our offices of 5 30,646,750

Problem:
We can notice that, the output doesn't contain ',' and 24/7

Expected Behavior
to find out more about how we use information, visit or contact any of our offices 24/7

Hi, How might I use symspell as a search algorithm that returns an index given a query string

Wolf,
I've updated the GitHub project that has my Levenshtein function. I've updated it to support using it via an instantiated class. Using it this way, it uses almost no memory, even temporarily, so it's memory impact is extremely negligible. I also made it as a nuget package, that is packaged as a .net standard 1.0 library. If you're interested in pulling it into symspell, let me know, and I can do that and submit a pull request. It would be cleaner I think, if the source was pulled into the symspell project, rather than create the package dependency. Before doing it though, it might be good if I had a better idea about how sysspell is used by you and other folks as far as multi-threading is concerned. The levenshtein distance can be done via an instantiated class, but the distance function itself is not threadsafe. There is also a static version of the function, that is only a tiny bit slower, but it loses the memory consumption advantage of the instantiated version.
Steve

Firstly, thank you for this is project - it looks really impressive.

I'm running this using the .Net Core on Linux and am running into problems with the capitalisation of files and folders when I try dotnet restore:

/opt/dotnet/sdk/2.1.4/NuGet.targets(227,5): warning MSB3202: The project file "/home/neil/Projects/SymSpell/symspell.Demo/symspell.Demo.csproj" was not found. [/home/neil/Projects/SymSpell/SymSpell.sln]

Having cloned the repo from Github, the folder is actually SymSpell.Demo with similar differences in the files within those folders (eg SymSpell.CompoundDemo SymSpell.Benchmark)

The fix isn't hard, either change the entries in SymSpell.sln to reflect the folders/filenames or adjust those folders/filenames to match the .sln file

I opted to change the folders and files but then had to adjust the location the demo looked for the frequency dictionary as that was looking in the capitalised version of the folder:

File not found: /home/neil/Projects/SymSpell/SymSpell/frequency_dictionary_en_82_765.txt

I am new to using .Net Core on Linux, so there may be some setting or tip I'm overlooking to help with this, but I suspect if the development is being done on Windows then the problem may have been unnoticed there because Windows tends not to be case sensitive for folders/files whereas Linux is.

Let me know if there are any more details you'd like me to supply for this. Thx!

At the moment, LoadDictionary assumes that the dictionary is located in a separate file on FS.
This limits ways of storing the dictionary.
E.g. resource or remote storage are out of options unless file is copied locally.

Alternatively all records could be parsed one by one, that however, would require writing usage code, which is almost identical to the body of the LoadDictionary.

P.S. I'm happy with doing a pull request in case it is an acceptable change

Greetings Sir,
I'm working on spell checker for Urdu Language.
Tried with your algorithm it gives me great results.
Now,
Can you explain how symspell algorithm works?
Like,if I wanted to calculate the edit distance of LD manual I know what's the way,
So how to calculate deletion in symspell or sympound manually,or what's the procedure for calculating manually.

Plus,I didn't got the algorithm.i know there is some deletion instead on insertion and all.
But I don't have the proper understanding of the given algorithm.
I read the algorithm written by you on medium.

I am working off the java Port of Symspell here (https://github.com/gpranav88/symspell), but I think this error would affect this too?

But looking at the code, @line78 in the parseWords(text) function:

return Regex.Matches(text.ToLower(), @"[\w-[\d_]]+") .Cast<Match>() .Select(m => m.Value);

It seems to split words such as "shouldn't" into 2 words ("shouldn" and "t").

Maybe I am wrong here, but shouldn't that be

return Regex.Matches(text.ToLower(), @"[\w-[\d_']]+") .Cast<Match>() .Select(m => m.Value);

so that contractions are added to the dictionary?

How are you supposed to integrate SymSpell into a C# project?

I installed the package via NuGet into a .NET 4.6.2 solution.

 var symSpell = new SymSpell(initialCapacity, maxEditDistance, prefixLength);

This line produces an error in my project: Cannot create an instance of the abstract class SymSpell.

Strangely, that same line does not produce an error in the demo project.

As an aside, I had no issues with integrating SymSpellCompound, although the Correct method seemed to only parrot the input words or input text rather than suggest any corrections.

edit: Well, I went ahead and compiled the SymSpell solution, and included the DLL as a reference in my project. I guess that's how? What's the point of the NuGet package then?

edit: Well, the reason why SymSpellCompound wasn't suggesting anything was because the dictionary wasn't loaded, and Lookup wasn't throwing an exception.

I try SymSpell and it looks great. But one thinks I notice almost intermediately it bring dictionary file in my project (even if I do not use it). I understand it helps with a quick start but I strongly believe in real application most of the users make own. But even if not. I think it will be better to have NuGet package split to SymSpell.Core and SymSpell.Dic.En for example. For keeping compatibility SymSpell could be composed of these two packages (something like Microsoft.AspNetCore.App.

hi , see the title.
Is anything wrong?

I hope you don't mind, I ported your C# code to Ruby.

https://github.com/PhilT/symspell

I've only just got it working and it's late so committing what I have now. Let me know if you'd like me to put any more of the original project details/copyright etc in my port.

Thanks for sharing!

This is not really a bug report, sorry for using the issue tracker for this, but I'd like to see if someone has worked on this. Please point me to a better place if you know one:

Has anybody worked on using SymSpell with German hunspell dictionaries? German uses compounds, so you cannot just export a long list from the hunspell dictionaries and use them as input for SymSpell. The hunspell dictionary has special flags that indicate which words can be used in a compound, these would need to be considered somehow.

HI,

First of all thank you for library.

I just work on Swift version for library.
And can't understand line:

if ((prefixLength - maxEditDistance == candidateLen)
                        && (((min = Math.Min(inputLen, suggestionLen) - prefixLength) > 1)
                            && (input.Substring(inputLen + 1 - min) != suggestion.Substring(suggestionLen + 1 - min)))
                           || ((min > 0) && (input[inputLen - min] != suggestion[suggestionLen - min])
                               && ((input[inputLen - min - 1] != suggestion[suggestionLen - min])
                                   || (input[inputLen - min] != suggestion[suggestionLen - min - 1]))))
                    {

When will be changed min?
Only if prefixLength - maxEditDistance == candidateLen ?
Or if Math.Min(inputLen, suggestionLen) - prefixLength) > 1 ?

Thanks a lot!

Hello sir, thank you for that nice code. I am writing a paper about segmentation of word and fuzzy search of words for certain script. Can i get a detailed algorithm used for this code.

Hi,

Apologies for using the issue tracker to ask a question.

How do I add a replacement for gr8 to great?? Is there a way to augment/extend the existing dictionary frequency_dictionary_en_82_765.txt using python to include these replacements? if yes how?

I also tried the example: I like readying, writing and singing and I was expecting an outcome of I like reading writing and singing but it never changed anything apart from striping up the punctuations.

I then checked the frequency_distionary and found that readying is included in there as well. Do I have to take it out to get it to replace readying with reading?

Best wishes and great work!

I'm currently using both Hunspell and SymSpell as main spelling correction system. They works both ok, SymSpell works great (quality, performances, etc.) That said, I have a question about Norvig probabilistic Spell Checker, that I show up with a simple case.
In some romanized languages, there is not one-to-one relation from the source script language term to the english (romanized) language term. So given that you have the romanization of let's say Hindi, you will get more possible english words as destination. Now this is a typical output of such a system: 1 (Hindi) word -> N (eng) words.
Typically decide which of the N words is the best is done with algorithm like beam search, viterbi, etc., but there are a lot of cases where the indecision stays on.
Also in other case, we have eng (N) -> hi (M), so this function is not bijective at all.
Given that a Spell Checker have knowledge of all (most of) the words in a language, etc. and supposed I need context (like in this case) to go back from eng (N) -> hi (M), do you think that SymSpell or Norvig's probabilistic model could give a valid hint about the M choices (or the N in the opposite way)? What's your opinion on that?

Is there a python binding for SymSpell (c++) so we can run it from Python without losing performance?

Any c/cpp porting that works?

I have noticed that you have updated the code to version 5.0, but the Nuget package is still referencing 4.1.

With this setting only the original word is returned in the output.
With MaxEditDistance = 1 the results are segmented but with results having an edit distance of 1.

Hi,
First of all SymSpell is damn fast and kind of does my job for spell correction but the issue I am facing is when my application user intentionally type any completely wrong word or sentence Symspell would would come up with a right word for it which can be avoided
Example
User types: avedoamlkejuike...
Syspell: a video am like juice keen...
Something like this which is totally irrelvant for my usecase
So how can I solve this just by using Symspell??
Thanks in advance

I have a doubt how the code for word segmentation will be able to segment the given example:

Input : thequickbrownfoxjumpsoverthelazydog
Output : the quick brown fox jumps over the lazy dog

because, in outer loop "j" is iterating from 0 to "input.length" and in inner loop "i" is varying from 1 to "imax"
assuming maxSegmentationWordLength is large enough and imax is alwayas taking the value (input.length - j)
so, as j is increasing imax is decreasing and the scope of substring that we take i.e. "part" will reduce. so, my concerns are

• the moment j crosses (input.length / 2), "i" would always be smaller than j and "part" substring would make no sense as we are taking part = input.Substring(j, i) and j > i
• other concern is strings that are at the other half of the middle index will not get segmented because they will never be assigned to "part" because of the above concern.
• So, how would we be able to segment complete string.
  I have not implemented the actual C# code, but wrote a python2 implementation of the same, and facing functional issues, which i have described above.

Please assist here. Thanks a lot.

Does SymSpell have a way to get the next valid set of letters given a prefix? E.g. if I give it "carpo" it would return "o" and "r" for "carpool" and "carport" (maybe others).

In SymSpell.CompoundDemo (v6.1, and v6.3),
in te dhird qarter oflast jear he hadlearned ofca sekretplan y iran is corrected to
in the third quarter of last year he learned of a secret plan a iran instead of
in the third quarter of last year he learned of a secret plan by iran as shown in README.md. Note that the y in front of iran is corrected to a instead of by.

May I know if I need to change some argument values to get the correction shown in README.md?

Dear Wolfgarbe,

I'm currently trying to make this program work for The Georgian language, I made a Dataset of words with their respective frequencies. For some reason, SymSpell does not return my suggestions. I'm sure that it is not SymSpell's fault. I attached the dataset.
The only difference I see between this and the Demo dataset is that mine is not sorted. So does the sort order matter in this case? or is there some other issue with my dataset?
workfile.txt

Sincerely

I am testing this library and need some advice as to whether this is the right tool for the job as the Top suggestion appears to be based simply on nearest match rather than any phonetic matching use by other spell checkers.

For example, searching "kween" returns a top match of "keen". Using Closest returns keen, tween, ween yet something like Hunspell will return Queen as the top match which is what I would expect in this case.

Hi,
I want to use symspell in android but couldn't find the dependencies to run it in android. How can I bind it with my android app?

Thanks.

Using SymSpell is great. However, I notice that it's all one static class. This makes it difficult to modify settings for different uses, especially the dictionary and verbosity. It can be helpful to enable this kind of code

SymSpell spellingDict1 = new SymSpell(path1, "", 0, 1, 0); // dictionary 1 with verbosity 0 SymSpell spellingDict2 = new SymSpell(path1, "", 0, 1, 1); // dictionary 1 with verbosity 1 SymSpell spellingDict3 = new SymSpell(path1, "", 0, 1, 2); // dictionary 1 with verbosity 2 SymSpell spellingDict4 = new SymSpell(path2, "", 0, 1, 0); // dictionary 2 with verbosity 0

the constructor could take the dictionary, or set it later, and optional addional parameters of verbosity, edit distance, lp, etc.

Great library! It works very well and is highly performant. To be honest I don't know or understand the underlying algorithm, but I have a suggestion / request from a user's perspective.

Right now the results seem only ordered by the # of character modifications, and are agnostic to the type of modification. However, this results in some unexpected "corrections". I think adjacent letter swaps should be highest priority, followed by missing a repeated letter, followed by add a letter, followed by remove a letter, followed by total letter replacement (this is most likely to result in a different intended word). Alternatively some sort of ranking/sorting involving for including more % of characters in the input word.

Examples:
basicly --> basic --> expected basically
collegue --> college --> expected colleague
finaly --> final --> expected finally
jist --> list --> expected gist (this one could be theoretically helped by the j sound being the same as g sound)
liase --> laser --> expected liaise
peice --> price --> expected piece
politican --> political --> expected politician
realy --> real --> expected really
rember --> member --> expected remember (this is two steps away, so maybe ignore it)
seige --> beige --> expected siege
tonge --> lounge --> expected tongue

This is an incomplete example list that I just got with a quick list from https://en.oxforddictionaries.com/spelling/common-misspellings
and using SymSpell.editDistanceMax = 3; (because 2 missed too many misspellings).

Overall, great library, thank you for maintaining it.

At the moment we're forced to writing our own wrappers around SymSpell that would allow mocking the class in our unit tests.
A predefined interface (or at least virtual attributes on most of the compute-intensive methods) would remove this requirement.

P.S. I'm happy with doing a pull request in case it is an acceptable change

When you load the dictionary all at once, this can consume quite a bit of memory and is a blocking process. If the dictionary is large, this can cause responsiveness issues, assuming you don't handle that another way.

What do you think about loading the dictionary in chunks in a Lookup method? So, that alternative Lookup method would load a chunk of the dictionary, try to find a match/suggestion in that chunk, and repeat until either a match/suggestion is found or there are no more chunks to load. Effectively, you'd be streaming in the dictionary as needed, rather than loading the whole shebang into memory.

I haven't looked at the source, so I don't know if that's feasible. Maybe you need the whole dictionary loaded? That said, an index-based approach instead of chunking might even be better, like Sphinx does.

Just some thoughts.

Hi,
This is very efficient and magical way for auto-correction.
Is it possible for you to provide Objective-c or Swift language version for the same?

It would be very helpful for me in my application.

Thank you.

Hi,

Is there a way to use symspell from the command line? I am not a programmer and want to use symspell without having to build a project (or use old ports). I am on linux. Thanks for any advice.

First of all, thank you for developing this tool. It is amazingly useful and fast!
I have been using the command line version since my programming skills are limited (my skills being in vba, sql and some java). I do have visual studio 2017 installed so perhaps that could help if I need to modify the project on my end..

So my hope would be that I could find a way to do the following:

1. process the frequency dictionary once and save it for re-use
   (I want to confirm that there is no way to have the processed dictionary be loaded into a db like mysql..I assume this won't work because mysql cannot create the proper indexes..correct?)

2. use a command line switch to set the number of matches returned (in frequency order of course) for edit distance 1 and edit distance 2 etc. So let's say I want the "top 5" and I set my max-edit distance to 3 then I would get 15 results (assuming there are >=5 matches for each edit distance). As it is now I may get a few for distance 1, a lot for edit distance 2 and a massive list for edit distance 3. I have been attempting to cleanup names from the census with have transcription errors and many times the correct name is the 1st or 2nd result in edit distance 2 or 3 (not edit distance 1). If I could get the top few matches from each edit distance then I have a phonetic algorithm that narrows the results.

3. ultimately I would really want to create an excel function that could call upon the command line for matches where the processed dictionary is already loaded into memory and that environment is accessible to vba

As a first step - items 1 and 2 are most important (saving the processed dictionary and setting the max # matches ordered by frequency)

Do you think this is possible? and could the dictionary ever be moved into a db? Thanks for your help and for sharing this excellent tool.

As a side note, could this ever be successfully migrated to nodejs to create an api?

I don't get the whole SuggestionStage thing. Comments say it's to help speed things up and be more memory efficient, but looking at the code I can't see how it would do anything but the opposite. Creating and populating the staging might be faster, but in the end it still has to go in the final data structure. So it's just an intermediate data object. I must be missing something.

I look at the current benchmark inside the project and I think it will be much better using BenchmarkDotNet which are used even by Microsoft for benchmarking dotnet core.

Hi,

1 - I want to add more columns like 'category' or 'type' or 'Culture' in the dataset and in that case maybe i need to have a word twice in the dataset.
for adding more clolumns which you mentioned it's possible, should I change the LoadDictionary method to support more than 2 columns ?

2 - what can I do for space seperated words, something like Mercedes benz ?

Best,
Amir

The Compound finds single spaces, Segmentation finds multiple spaces.

For the case when many words are concatenated without spaces,
how is Segmentation implemented in Python to add spaces where needed?

Is there a way to exclude nouns while correcting for spelling mistakes. When I do a spell check for a paragraph, the names also get changed into some word in the dictionary. Please let me know if there are any options available to implement this.

Currently I am trying to figure out an approach to make a spell checker for Central Kurdish. Just like some other Indo-Europian languages, Kurdish language has Clitic pronouns. It's a bit tricky because:

• Kurdish has two sets of clitic pronouns that are used depending on the tense of the verb and whether it's transitive or not.
• The pronouns can stick to most parts of speech: Nouns, Verbs, Adjectives and Adverbs.
• They don't always stick to the end of the word, they can also appear in the middle of the word (after the root of the verbs or after the first word of a compound word)
• There can be more than one pronoun stuck to a verb (One of them acts as a subject the other acts as an object). See example 3

Because there are two sets of pronouns, pre-calculating a dictionary with 100K words would results in about 1M words. That's before calculating the edit distances.

Which of these approaches do you think is the best in this case?

• Compile a big dictionary with every valid combination
• Compile a smaller dictionary composed of only the base words and categorize them and then expand each of them at run-time based on the PoS and other properties of each word.
• Use an Recurrent Neural Network to do the job

Examples:

Note: letters in bold are pronouns.

Eat

• I eat => Min Dexom [Min, m]
• We eat => Ême Dexoyn [Ême, yn]
• They eat => Ewan Dexon [Ewan, n]

Work

• I worked => Min Karmdekird [Min, m]
• He Worked => Ew Karîdekird [Ew, î]
• They worked => Ewan Karyandekird [Ewan, yan]

Forgive

• [You (plural)] Forgive me => Bimbexshin [m, n]
• [You] Forgive her => Bîbexshe [î, e]
• [You] Forgive them => Biyanbexshe [yan, e]

I'm trying to use SymSpell on a non-English text (Norwegian). Got a good database from https://github.com/hermitdave/FrequencyWords/blob/master/content/2018/no/no_full.txt.

However, the following example does not work at all:

//lookup suggestions for multi-word input strings (supports compound splitting & merging)
inputTerm="dettefungererikkeveldigbra";
maxEditDistanceLookup = 2; //max edit distance per lookup (per single word, not per whole input string)
suggestions = symSpell.LookupCompound(inputTerm, maxEditDistanceLookup);

//display suggestions, edit distance and term frequency
foreach (var suggestion in suggestions)
{ 
  Console.WriteLine(suggestion.term +" "+ suggestion.distance.ToString() +" "+ suggestion.count.ToString("N0"));
}

Not sure if this is an error on my part, or if non-English is not supported.

I would have wished the output to be "dette fungerer ikke veldig bra", but instead I am returned with the original input.

Lookup() however, works I think.

I'm not sure if SymSpell already has support for weighted edit distance. If so, please tell me how to use it.

Otherwise, I suggest to add this as another possible distance metric, in addition to Levenshtein and Damerau-Levenshtein. The implementation itself shouldn't be problematic: just use the weight matrix instead of the default unit cost. The matrix is input to the constructor, and for command line use it can be stored in a file. (I could in principle do it myself, but I don't know C#)

if I search "whatareyou" it is giving "what you"

Using frequency_dictionary_en_500_000.txt, I spotted some errors:

• youre should be you're
• dont should be don't
• dont's should be don'ts
• Lots of weird [a-zA-Z]+[0-9]+ words, too.

This list seems to include common misspellings and 1337 speak (e.g., f1nancially, di3.)

Thank you for this great code. I'd like to ask you about recommendation/suggestion on how I could make loading a big dictionary faster and not slowing down the application.
I have an idea, please answer as a creator if this is going to work in your opinion:

1. I thought about adding words in smaller chunks via multiple for loops using CreateDictionaryEntry() function on arrays.
2. To make this not stall the application, I could run those loops on a background thread. Small enough chunks could not affect performance, while gradually building a dictionary before a user gets to type in a word.

My previous implementation was my own, but it was using SQL database. Queries were too slow for this kind of usage or too many with a bigger dictionary.

An issue - for some reason, CreateDictionaryEntry() is the only way for me that works after using LoadDictionary() first. CreateDictionary() doesn't add more terms to an existing database. Word count stays the same. Maybe it's because it's plain text without a second column for word frequency?

And two short questions:

1. How to add multi-word phrases to dictionary among normal single word ones? Currently, CreateDictionaryEntry() seems to add some phrases while throwing others out. If I start with a number and a second word has only letters, it shows up in Lookup(). But if the first one starts with an uppercase letter and second is just 3 uppercase letters, Lookup() doesn't work. The same when there's some punctuation.
2. How to add phrases (single and multi-word) with punctuation and make them show up as suggestions among single word phrase?

Currently there's no support for various encoding style like ISO-8859-1 while reading data to create Dictionary.
Having Encoding as optional argument will help load Accented characters also like å

if I type hekko (for hello) ; rppm (room) - it is giving me unexpected results. it is not giving me hello and room. (Edit distance = 2 and Verbose = ALL)

Excuse me, but the link points to the dictionary file you use https://github.com/wolfgarbe/SymSpell/blob/master/symspell/frequency_dictionary_en_82_765.txt is reported 404 not found now. Would there be anywhere so I could find the dictionary? Thanks!

This question might sound very basic/stupid but I am very new to c# and. net , I installed VS code and opened the downloaded symspell project code in it. Now how can i run the demo ?

In problems it shows the error:- The type 'List<>' is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. [SymSpell.Test]

When i try to run the SegmentationDemo by ctr+f5 it gives error :-

Unhandled Exception: System.PlatformNotSupportedException: Operation is not supported on this platform.
at System.ConsolePal.set_WindowWidth(Int32 value)
at symspell.Benchmark.Benchmark.Main(String[] args) in /Users/aashishamber/Downloads/SymSpell-master/SymSpell.Benchmark/SymSpell.Benchmark.cs:line 52

Kindly help how to run the code.

Is there a way to lookup a composite word but with Verbosity param?
For example I have names of people and I would like to get back the closest 5 people, not just the best match.