Detect files whose encoded has been corrupted by a text editor ? about uchardet HOT 1 CLOSED

No uchardet is made to detect correctly coded files. I don't think that it should be its role to detect broken encodings. Actually even if just considering the cases of single corruption (not reopened again and again with other encodings), we would get nb-of-encoding*nb-of-encoding cases.
Just for this reason, I would reject the feature request. uchardet should remain as simple as possible in its use cases. I am thinking about adding language detection since the algorithm is already doing it anyway (so we are already half the way), but that's about it.

Finally I am not even sure how feasible this would be with statistical algorithms. It would require generating statistics about every couple of encoding with datasets of files decoded by error in this given couple (actually even a 3-tuple because language matters), so that we get statistics data of characters and character sequences. Well not sure where to get this data reliably so we'd have to basically build it ourselves. That's a lot of work (that's already a lot of work to make data for every (lang, charset), now we'd have to get it for every (lang, charset1, charset2)?) for a very uncertain result.
So I'm thinking that uchardet algorithm is very efficient for charset detection, but I'm not sure how efficient it would be in charset decoding error detection. I guess someone would have to try. But it may turn out that another algorithm would be a better suit.

So thanks for the proposition, but I will close this feature request now.

from uchardet.