berkerpeksag / astor Goto Github PK
View Code? Open in Web Editor NEWPython AST read/write
Home Page: https://pypi.org/project/astor/
License: BSD 3-Clause "New" or "Revised" License
Python AST read/write
Home Page: https://pypi.org/project/astor/
License: BSD 3-Clause "New" or "Revised" License
From https://travis-ci.org/berkerpeksag/astor/jobs/357390920:
======================================================================
FAIL: test_convert_stdlib (tests.test_rtrip.RtripTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/build/berkerpeksag/astor/tests/test_rtrip.py", line 24, in test_convert_stdlib
self.assertEqual(result, [])
AssertionError: Lists differ: ['/home/travis/build/berkerpeksag/astor/.t[35785 chars].py'] != []
First list contains 344 additional elements.
First extra element 0:
'/home/travis/build/berkerpeksag/astor/.tox/py37/lib/python3.7/os.py'
I have installed the 0.5
version of astor
with pip install astor
, I can see it in my virtual environement when I do pip list
. However, if I try to import astor
then ModuleNotFound
is thrown. What am I doing wrong?
Currently, the only command line supported is python -m astor.rtrip [readonly] [<source file or directory>]
Locally, I've enhanced my own copy of rtrip
with a few more options, but the way I did that was, uhhhh, non-standard.
When I started this project, I had some Python interpreter where -m <module>
ate any subsequent flags on the command line, e.g. -r
didn't work. I don't know what that was, haven't found it again, can't recreate it, don't care.
I also have to confess that I don't know which command line argument parser is winning the hearts and minds of astor-lovers.
So, I have some functionality I'd like to add in a branch, but I haven't created a PR because I assume it's just really wrong code.
I don't know if we want to add a __main__.py
and be able to support python -m rtrip ...
as well as other options, or whether we just want to do a better job of doing the astor.rtrip
command line.
If anybody could help, that'd be awesome.
I can take a stab at this when I get some time, but posting this now in case someone else is interested.
Sometimes I pass in an incorrect AST to an astor method, e.g. astor.to_source(ast)
(e.g. ast contains a raw int
instead of wrapping it an ast.Num
). This leads to tricky errors internally in astor that can be hard to understand for the user (this particular error resulted in a failed assertion in precedence_setter.set_precedence
).
I propose that astor validates the input AST for any user facing API functions. This would allow any internal logic to assume that the AST is valid and for users to be immediately notified that the input AST is wrong, rather than having to parse errors or exceptions from the internal logic.
We're missing a proper testing suite.
I ran into an issue where I defined a custom AST
node, and when I tried rendering the source with astor.to_source(node)
, it crashed saying No defined handler for node of type xxx
.
I understand that astor
would not know how to render some unknown AST
node, but it would be nice if we could register a visit function, such that when astor
encounters a unknown AST
node, instead of crashing it checks the registry for the visit function.
There are at least two reasons for having a custom AST
class:
AST
class.AST
class, e.g. a Attribute
. Because obj.__class__.__name__
is used, even though it inherits from some known AST
class, astor
does not know what to do with it.I understand why the first use case would not be convincing. But number two should be common enough that allowing to register one AST
class name as an alias for another, or alternatively allowing to register an explicit visit function would be useful (or both).
My use case is as follows: I do the tree walk and then replace some nodes with a proxy node. The proxy node refers to the original node. However, later at my convenience, I replace the node the proxy refers to with another node. This allows me to switch out nodes without having to walk the tree. Currently, I monkey patched astor
to be able to handle this as follows:
def visit_ASTNodeRef(self, node, *largs, **kwargs):
return self.visit(node.ref_node, *largs, **kwargs)
SourceGenerator.visit_ASTNodeRef = visit_ASTNodeRef
But, it would be nice if there was an official way of doing this. I can work on a PR if that sounds acceptable.
I am facing a subtle problem when a big float literal is converted to infinity by Python. This is illustrated in the following snippet (tested with astor 0.5 on Python 3.4):
print (astor.to_source(ast.parse('a = 1e400')))
# --> 'a = inf'
print (astor.to_source(ast.parse('a = -1e400')))
# --> 'a = (- inf)'
These round-trip calls are supposed to produce equivalent Python code that is syntactically valid. This is not the case here since inf
is not a builtin object of Python. Actually, the bug occurs because the infinity value is already in the ast node; I assume that astor.to_source
calls repr
on the value carried by this node, which indeed returns the string 'inf'
. A satisfying solution for me could be this:
print (astor.to_source(ast.parse('a = 1e400')))
# --> "a = float('inf')"
print (astor.to_source(ast.parse('a = -1e400')))
# --> "a = - float('inf')"
I think it would be useful to have https://github.com/berkerpeksag/astor/blob/master/astor/misc.py#L185-190 as an helper in astor.
This really is not an issue but is there a way to add a pretty print format to codegen that will handle outputing docstrings of this form
"""
This
is a pretty
docstring!
"""
See hylang/hy#824 (comment) for more info.
Move its content to astor.utils
or something else.
Hi, our builds started failing with AttributeError: module 'astor' has no attribute 'codegen'
:
https://travis-ci.org/google/tangent/jobs/300437272
I wonder if it's related to the new 0.6.1 release?
When creating a FunctionDef
ast, if you have a Call
as part of the body, the Call
will not be on a new line.
def foo():
bar()
will appear as
def foo():bar()
Not sure why and no time to debug right now.
sudo python setup.py install will appear to work under anaconda, but will install into system python directories.
All that setup stuff is a complete mystery to me.
I'd like to make a new Hy release soon (before August), and Hy now depends on new features and bug fixes in astor, particularly for Python 3.7 compatibility.
Running in Python 3.6.4
f"Hello " \
" {name}"
# printing the above yields
# "Hello {name}"
src dump
Module(body=[Expr(value=JoinedStr(values=[Str(s='Hello {name}')]))])
dst dump:
Module(
body=[
Expr(
value=JoinedStr(
values=[Str(s='Hello '), FormattedValue(value=Name(id='name'), conversion=-1, format_spec=None)]))])
Will investigate why this happens
Python 3.4 introduced a new node of type NameConstant
for True,False & None.
Python 3.4.1rc1 (default, May 5 2014, 14:28:34)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import astor
>>> import ast
>>> astor.to_source(ast.parse("None"))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/codegen.py", line 40, in to_source
generator.visit(node)
File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/misc.py", line 155, in visit
return visitor(node)
File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/codegen.py", line 490, in visit_Module
self.visit(stmt)
File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/misc.py", line 155, in visit
return visitor(node)
File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/codegen.py", line 179, in visit_Expr
self.generic_visit(node)
File "/usr/lib/python3.4/ast.py", line 255, in generic_visit
self.visit(value)
File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/misc.py", line 155, in visit
return visitor(node)
File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/misc.py", line 149, in abort_visit
raise AttributeError(msg % node.__class__.__name__)
AttributeError: No defined handler for node of type NameConstant
Somewhat related to #82, the codegen
of astor 0.5 would produce nan
if given a Num
node that contained NaN. In astor 0.6, the following raises an AssertionError
in code_gen.py:
import ast, astor
nan = 1e1000 - 1e1000
print(astor.code_gen.to_source(
ast.Module([ast.Expr(ast.BinOp(
ast.Num(1.0),
ast.Add(),
ast.Num(nan)))])))
While Python has no NaN literal, you could represent ast.Num(nan)
as an expression (such as 1e1000 - 1e1000
) if you want to generate real Python. See hylang/hy#1447.
Passing a method to code_to_ast
causes the following error.
>>> import astor
>>> from requests.auth import HTTPDigestAuth
>>> node = astor.code_to_ast(HTTPDigestAuth.build_digest_header)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\miau\Documents\repo\astor\astor\file_util.py", line 102, in __call__
return cache[key]
KeyError: ('C:\\ProgramData\\Anaconda3\\lib\\site-packages\\requests\\auth.py', 128)
It seems that astor doesn't look into ast.ClassDef
.
Line 99 in 0a02019
Here is an example in the 3.5 std library. Perhaps we can find/reuse the std library code for handling the file encoding descriptor?
>>> import astor >>> astor.parsefile('/usr/local/lib/python3.5/sqlite3/test/dbapi.py') Traceback (most recent call last): File "", line 1, in File "/home/pat/projects/opensource/astor/astor/misc.py", line 166, in parsefile fstr = f.read() File "/usr/local/lib/python3.5/codecs.py", line 321, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 120: invalid continuation byte
Someone who knows sphinx better than me (e.g. at all) should look into this.
changelog.rst will have to be modified.
Look at modifications to index.rst to see how to make it still display properly at github.
The following code won't work in Python 3.7:
Line 311 in 5f189fc
Full list:
astor/code_gen.py:311: def visit_FunctionDef(self, node, async=False):
astor/code_gen.py:312: prefix = 'async ' if async else ''
astor/code_gen.py:325: self.visit_FunctionDef(node, async=True)
astor/code_gen.py:367: def visit_For(self, node, async=False):
astor/code_gen.py:369: prefix = 'async ' if async else ''
astor/code_gen.py:376: self.visit_For(node, async=True)
astor/code_gen.py:383: def visit_With(self, node, async=False):
astor/code_gen.py:384: prefix = 'async ' if async else ''
astor/code_gen.py:395: self.visit_With(node, async=True)
The new f-strings are only single-quoted for now, which should work fine, but which will look ugly in some cases.
Adding the ability to fix those will involve code changes to code_gen.py and string_repr.py.
This is a science fair project I am unwilling to undertake at the current time.
Although the AST does not include a Comment node, it is frequently a good idea to insert comments into generated code. It's possible to wrap textual comments in a string instead, but it doesn't look as nice.
I've been experimenting and it looks like it should be feasible to add a custom Comment node (inheriting from ast.Expr, possibly).
The really interesting bit is when you allow the Comment's value to be not just strings, but other AST nodes. In this case you could generate syntactically-valid and properly indented commented-out code.
I've got this working for single line statements and I think I can do it for multiple line blocks like function definitions as well. However, before I go further I want to be sure something like this would be accepted into the project.
$ python -c 'import ast, astor; print(astor.code_gen.to_source(ast.Module(body = [ast.parse("def f():\n \"docstring\"\n 1")])) )'
def f():
1
$ python -c 'import ast, astor; print(astor.code_gen.to_source(ast.Module(body = [ast.parse("class C:\n \"docstring\"\n 1")])) )'
class C:
1
$ python -c 'import ast, astor; print(astor.code_gen.to_source(ast.Module(body = [ast.parse("\"hello\"\n1")])))'
1
Right now, codegen.py just says "distributed under the BSD license", but there is no such thing as "the BSD license". Is it 2-clause? 3-clause? 4-clause?
Maybe @treo (author of the original gist) can clarify what the original license was supposed to be?
See also https://en.wikipedia.org/wiki/BSD_licenses for reference.
Note also that the rest of the astor codebase is under the BSD 2-clause (https://en.wikipedia.org/wiki/BSD_licenses#2-clause_license_.28.22Simplified_BSD_License.22_or_.22FreeBSD_License.22.29).
node.docstring
has just removed from 3.7.0.b5 due to backwards compatibility reasons. See python/cpython#7121 for details.
By looking at the pull request, reverting 524d525 might be enough.
@radomirbosak would you like to submit a pull request?
Hello,
There seems to be an error turning an ast back into the correct form when expanding tuples. For example, take the following code.
a = (1, 2)
b = (*a, 3)
Running rtrip
on this code fragment leads to the following
a = 1, 2
b = *a, 3
which unfortunately is not valid python.
Looks like a test-only failure, though.
======================================================================
FAIL: test_convert_stdlib (tests.test_rtrip.RtripTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/build/python-astor/src/astor-0.6/tests/test_rtrip.py", line 24, in test_convert_stdlib
self.assertEqual(result, [])
AssertionError: Lists differ: ['/usr/lib/python3.6/test/test_fstring.py'[34 chars].py'] != []
First list contains 2 additional elements.
First extra element 0:
'/usr/lib/python3.6/test/test_fstring.py'
+ []
- ['/usr/lib/python3.6/test/test_fstring.py',
- '/usr/lib/python3.6/idlelib/grep.py']
python -c 'import ast, astor; print(astor.code_gen.to_source(ast.Set([])))'
{}
Since Python has no literal syntax for the empty set, and we can't use set()
because the name set
might be rebound, I think the right representation is {1}.__class__()
.
It looks like astor is not ready for some of the newer Python features.
Python 3.6.1
>>> import astor, ast
>>> astor.__version__
'0.5'
>>> compile("{1: 1, **{2: 2}}", "<str>", "exec", ast.PyCF_ONLY_AST)
<_ast.Module object at 0x00000180BB3F1F28>
>>> astor.codegen.to_source(_)
'{1: 1, None: {2: 2, }, }'
The stars are None:
.
I get the following backtrace using astor 0.5 on python 3.5
File "/Users/sborini/Work/github/simphony/simphony-remote/venv/lib/python3.5/site-packages/traitlet_documenter/util.py", line 62, in get_trait_definition
return astor.to_source(node.value).strip()
File "/Users/sborini/Work/github/simphony/simphony-remote/venv/lib/python3.5/site-packages/astor/codegen.py", line 40, in to_source
generator.visit(node)
File "/Users/sborini/Work/github/simphony/simphony-remote/venv/lib/python3.5/site-packages/astor/misc.py", line 161, in visit
return visitor(node)
File "/Users/sborini/Work/github/simphony/simphony-remote/venv/lib/python3.5/site-packages/astor/codegen.py", line 366, in visit_Call
self.conditional_write(write_comma, '*', node.starargs)
AttributeError: 'Call' object has no attribute 'starargs'
The last release on PyPI is from 2015-04-18. There have been quite a lot of commits since then. It would be great not having to install directly from Github.
Unfortunately, the Sphinx guys (ab)used the docutils class::
keyword for Python classes.
That means that the doc looks fine here but the class names are missing here.
A couple of years ago, I submitted an issue and PR to github to be able to support such things. They finally accepted it last month; the result is that you can make a doc that displays nicely under Sphinx, and reasonably at github, and also explains you should view it elsewhere. That is all documented here
So the right answer is to update index.rst
accordingly.
The profiler shows some low-hanging fruit for speedups. I'll spend a couple of hours looking at what I can do without going overboard.
I will be committing a branch with tests to deal with these:
Keyword-only should arguments come before '**'
'from .. import' with no trailing module name did not work.
try/finally needs to come after else
Sometimes integers have attributes, e.g. (0).real We need the parentheses in this case, and did not have them.
When yield is an expression rather than a statement, it can be a syntax error if it is not enclosed in parentheses.
Related to #82 and #85, ast.Num
s containing non-finite complex numbers will be dumped as their reprs instead of legal Python expressions that evaluate to the right floating-point constructs.
import ast, astor
x = (1e1000 - 1e1000) - 1e1000j
print(x)
print(astor.code_gen.to_source(ast.Module([ast.Expr(ast.Num(x))])))
Both lines print as (nan-infj)
.
It was added in f87ae3b by @radomirbosak. We need to document it in docs/index.rst
and docs/changelog.rst
.
However, I don't consider this an urgent issue and it may be better to wait at least one release in order to avoid potential backwards-compatibility issues. Or we could just add a note in docs/changelog.rst
and declare it as 'provisional'.
Hi guys!
Found an issue with astor when working with non printable chars in string. For example code below yields an error:
old_source="""
def write(self, param):
if param=='\n':
return;
"""
tree = ast.parse(old_source)
new_source = to_source(tree)
print(new_source)
Error:
if param=='
^
SyntaxError: EOL while scanning string literal.
And thanks for astor, it's really useful!
Hi!
I have the following example:
import astor
import ast
class CallWrapper(ast.NodeTransformer):
def visit_Call(self, node):
if node.col_offset == 0:
return None
return node
src = open('test.py', 'r').read()
p = ast.parse(src)
CallWrapper().visit(p)
ast.fix_missing_locations(p)
new_src = astor.to_source(p)
print(new_src)
It works with Python 2.7, but raise the following error on Python3.5
Traceback (most recent call last):
File "easy.py", line 15, in <module>
new_src = astor.to_source(p)
File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/code_gen.py", line 51, in to_source
generator.visit(node)
File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/node_util.py", line 137, in visit
return visitor(node)
File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/code_gen.py", line 662, in visit_Module
self.write(*node.body)
File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/code_gen.py", line 156, in write
self.visit(item)
File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/node_util.py", line 137, in visit
return visitor(node)
File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/code_gen.py", line 260, in visit_Expr
set_precedence(node, node.value)
AttributeError: 'Expr' object has no attribute 'value'
Am I missing something? Is it a bug? If so, please if possible, point me some direction that I'd love to create a PR.
Thank you
In[15]: astor.to_source(ast.parse('1+2'))
Out[15]: '(1 + 2)'
These are not present in 0.6, but present in 0.6.1 and latest 0.6.2 release:
======================================================================
ERROR: test_codegen_as_submodule (tests.test_misc.PublicAPITestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/build/python-astor/src/astor-0.6.2-py2/tests/test_misc.py", line 39, in test_codegen_as_submodule
with self.assertWarns(DeprecationWarning) as cm:
AttributeError: 'PublicAPITestCase' object has no attribute 'assertWarns'
======================================================================
ERROR: test_codegen_from_root (tests.test_misc.PublicAPITestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/build/python-astor/src/astor-0.6.2-py2/tests/test_misc.py", line 28, in test_codegen_from_root
with self.assertWarns(DeprecationWarning) as cm:
AttributeError: 'PublicAPITestCase' object has no attribute 'assertWarns'
----------------------------------------------------------------------
Not sure if it's introduced by Python 3.5, though. Hope it helps:
.....E.
======================================================================
ERROR: test_try_expect (tests.test_codegen.CodegenTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/build/python-astor/src/astor/tests/test_codegen.py", line 47, in test_try_expect
self.assertAstSourceEqual(source)
File "/build/python-astor/src/astor/tests/test_codegen.py", line 24, in assertAstSourceEqual
self.assertEqual(astor.to_source(ast.parse(source)), source)
File "/build/python-astor/src/astor/astor/codegen.py", line 40, in to_source
generator.visit(node)
File "/build/python-astor/src/astor/astor/misc.py", line 161, in visit
return visitor(node)
File "/build/python-astor/src/astor/astor/codegen.py", line 493, in visit_Module
self.visit(stmt)
File "/build/python-astor/src/astor/astor/misc.py", line 161, in visit
return visitor(node)
File "/build/python-astor/src/astor/astor/codegen.py", line 288, in visit_Try
self.visit(handler)
File "/build/python-astor/src/astor/astor/misc.py", line 161, in visit
return visitor(node)
File "/build/python-astor/src/astor/astor/codegen.py", line 300, in visit_ExceptHandler
self.body(node.body)
File "/build/python-astor/src/astor/astor/codegen.py", line 99, in body
self.visit(stmt)
File "/build/python-astor/src/astor/astor/misc.py", line 161, in visit
return visitor(node)
File "/build/python-astor/src/astor/astor/codegen.py", line 179, in visit_Expr
self.generic_visit(node)
File "/usr/lib/python3.5/ast.py", line 255, in generic_visit
self.visit(value)
File "/build/python-astor/src/astor/astor/misc.py", line 161, in visit
return visitor(node)
File "/build/python-astor/src/astor/astor/codegen.py", line 366, in visit_Call
self.conditional_write(write_comma, '*', node.starargs)
AttributeError: 'Call' object has no attribute 'starargs'
----------------------------------------------------------------------
Ran 7 tests in 0.012s
FAILED (errors=1)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.