x1angli / cvt2utf Goto Github PK

Python 3.6.1
pip install convert2utf
from cvt2utf import Convert2Utf8

    from cvt2utf import Convert2Utf8
ModuleNotFoundError: No module named 'cvt2utf'

We suppose it's straightforward to detect the charset of a non-UTF text file -- but it's not. When using 3rd-party tools to guess the original encoding method of a file, we don't have the author to ask, we don't have the context of the text file -- the entire process feels like reverse-engineering... In short, we rely on probability models to detect the charset, without any guarantee

As a result , we have to set a minimum threshold. Only those encoding results returned by chartdet that are above that threshold level would be accepted. Otherwise, if chartdet can't tell the encoding of a file with confidence, we would simple throw a warning message.

Thanks @runhwguo for your valuable proposal.

bytedata.decode 采用 latin_1 的方式，GB2312 编码的文件也能通过。

Describe the bug
A clear and concise description of what the bug is.
I use the tool to convert .cue and .log files.
I discovered problem with some UTF-8 encoded cue files and many UTF-16 encoded log files.
Some UTF-8 encoded files do not transcoding well and still have problem in mapping.
UTF-16 encoded log files are recognized as latin_1 encoded.

Example is attach.
.bak is the original fils.
error.zip

In virsual stduio, cpp,i want it's chatset can be utf-8-sig
and in java ,i want to be utf-8, without bom.

Python 3.5.4, win32

Python>pip install convert2utf
Collecting convert2utf
Using cached convert2utf-1.0.0.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "\AppData\Local\Temp\pip-build-lhz8yq_d\convert2utf\setup.py", line 12, in
long_description = pypandoc.convert('README.md', 'rst')
File "python\python35-32\lib\site-packages\pypandoc_init_.py", line 66, in convert
raise RuntimeError("Format missing, but need one (identified source as text as no "
RuntimeError: Format missing, but need one (identified source as text as no file with that name was found).