astor.parsefile does not handle non-UTF8 files properly about astor HOT 9 CLOSED

Yes, we should use io.open(..., encoding='utf-8') instead of https://github.com/berkerpeksag/astor/blob/master/astor/misc.py#L165

from astor.

I tried that -- it doesn't actually help. The problem is that the files in question are encoded in iso-8859-1, not utf. They are properly marked according to https://www.python.org/dev/peps/pep-0263/ with a first line of

#-*- coding: iso-8859-1 -*-

PEP263 gives the instructions for decoding this -- read the first two lines, look for a match for this regexp, etc.

1. We can do that -- it's not rocket science -- but I was wondering if there was an accessible thing in the library to do it already (I think the implementation of PEP263 does it down in the compiler.)

2. Then we have the round-trip issue. Obviously, we don't need to go back to any random encoding, but I think the default encoding for < Python 3.0 is ASCII, so we probably haven't thought out how to generate non-ASCII characters in the output dump -- we probably need to automatically add the UTF-8 encoding at the top of the file.

from astor.

1. We can do that -- it's not rocket science -- but I was wondering if there was an accessible thing in the library to do it already (I think the implementation of PEP263 does it down in the compiler.)

https://docs.python.org/3/library/tokenize.html#tokenize.detect_encoding would probably do the trick. We can backport it for Python 2.

from astor.

Great! I'll work on that as part of merging anti8.

from astor.

Why not use the tokenize.open?

from astor.

tokenize.open also doesn't exist on Python 2. Since detect_encoding accepts a file object's readline method, we'll need to do duplicate tokenize.open anyway. So I'm +1 for using tokenize.open.

from astor.

It turns out the only failing issues I had were either on 3.4.0, or with broken pretty-print string code. I added some goofy stuff to the pretty-print string code, and everything seems to work. If nobody has a counter-example, I will close.

from astor.

So, for some reason, Python 3.6 works on my home machine but not my work machine. Both Ubuntu using the same PPA.

from astor.

I'm pretty sure this works on Python 3 now.

Closing until someone who cares generates any sort of failing test.

from astor.