GB18030 file detected as WINDOWS-1252 about uchardet HOT 6 CLOSED

Hi,

On Wed, Jan 20, 2016 at 3:22 PM, Xijian Yan [email protected]
wrote:

I encountered an issue when I try to use uchardet to detect a file almost
all English(a very little Chinese)
The file is GB18030 encoded but detected as WINDOWS-1252

Could you send the file and make a bug report about it?
I'll have a look when I can. Thanks!

Jehan

—
Reply to this email directly or view it on GitHub
#23.

from uchardet.

@Jehan I'm home now. I will send it to you at work tomorrow. Thanks for concerns. :)

from uchardet.

my love gbk.txt
Hi, @Jehan .That's the file.

from uchardet.

Ok I had a quick look. It looks like the current implementation only even tries to detect GB18030 when the escape characters are used, which is not the case of your file. Apparently there are several ways to encode the same characters, sometimes with escape characters, sometimes not (from the few I can understand with a quick look on Wikipedia).

Also I guess that your file being mostly English, it does not help, statistically.

I'll need to have a closer look to the structure of this encoding later so that I can try and fix this. Thanks for the report!

from uchardet.

I read some articles. The reason might be using GBK or GB18030 characters in file encoded in GB2312. GB2312, GBK, GB18030 are all Simplified Chinese encoding. GB18030 is the latest.

from uchardet.

Moving bug reports to the new project hosting.
Work will happen there: https://bugs.freedesktop.org/show_bug.cgi?id=97521

from uchardet.