neizod / duppub Goto Github PK

right now i use edit distance to check similarity between 2 strings, it run so slow i had to limit string size. are there any better algorithm?

that i have in mind:

• approximate algorithms are fine!
• i need to check all pair, not just 2 strings.

Right now we have a manual test, which run by this in command line:

python report.py example_input.csv

Can we make them automate somehow? I mean the result is similarity percentage, and many algorithm report different percentage too. Any ideas?

Right now the script can be invoke with only one argument, the CSV file.

It'd be nice if we can invoke it and supply more arguments to tune the performance, such as

• Threshold (default: 80%) -- report similarity only above this threshold.
• Limit Characters (default: 100 chars) -- right now the algorithm run so slow we need to chop string to compare down, we should be able to tune this parameter.
• Algorithm (default: levenshtein) -- other slower algorithm available is edit_distant.

Technical info: now the main function doing the hard code parsing argument with sys.argv, I think Python ArgParse can do this job better & cleaner.