Code Monkey home page Code Monkey logo

typed_ast's Introduction

End of life

This project is no longer maintained.

Use the standard library ast module instead. See #179.

Typed AST

Build Status Chat at https://gitter.im/python/typed_ast

typed_ast is a Python 3 package that provides a Python 2.7 and Python 3 parser similar to the standard ast library. Unlike ast up to Python 3.7, the parsers in typed_ast include PEP 484 type comments and are independent of the version of Python under which they are run. The typed_ast parsers produce the standard Python AST (plus type comments), and are both fast and correct, as they are based on the CPython 2.7 and 3.7 parsers. typed_ast runs on CPython 3.6-3.10 on Linux, OS X and Windows.

Note: Starting with Python 3.8, we recommend to use the native ast parser (see below).

Development Philosophy

This project is a (mostly) drop-in replacement for the builtin ast module. It is intended to be bug-for-bug compatible and behave identically, except for the presence of a few additional fields on the returned classes and a few additional optional arguments to the parse call. Therefore, typed_ast will not accept any bugfixes for bugs in ast -- they should be fixed upstream instead. To avoid feature bloat, any new features for typed_ast should have the potential to be broadly useful and not be built just for one niche usecase or in a manner such that only one project can use them.

Incompatibilities

For the purposes of consuming syntax trees, this should be a drop-in replacement. It is not a drop-in replacement for users that wish to create or transform ASTs, as a number of syntax tree classes have additional fields that must be populated when constructing them.

Due to reliance on certain C APIs, this library does not build on and there are no plans to support PyPy.

Python 3.8

typed_ast will not be updated to support parsing Python 3.8 and newer. Instead, it is recommended to use the stdlib ast module there, which has been augmented to support extracting type comments and has limited support for parsing older versions of Python 3.

Submodules

ast3

The ast3 parser produces the AST from a Python 3 code, up to Python 3.7. (For rationale and technical details, see here.) The AST it currently produces is described in ast3/Parser/Python.asdl. If you wish to limit parsing to older versions of Python 3, ast3 can be configured to to give a SyntaxError for new syntax features introduced beyond a given Python version. For more information, see the module docstring in typed_ast/ast3.py.

ast27

The ast27 parser tracks the standard Python 2.7 AST, which is expected to never receive further updates. The AST it produces is described in ast27/Parser/Python.asdl. For more information, see the module docstring in typed_ast/ast27.py.

conversions

typed_ast also provides a conversions module which converts ast27 ASTs into ast3 ASTs. This functionality is somewhat experimental, however. For more information, see the py2to3 docstring in typed_ast/conversions.

Note: as these parsers consider type comments part of the grammar, incorrectly placed type comments are considered syntax errors.

Releases

To make a new typed_ast release, see release_process.md.

typed_ast's People

Contributors

ben9923 avatar cameronnemo avatar cclauss avatar danielmenzel avatar ddfisher avatar ethanhs avatar expobrain avatar freundtech avatar gvanrossum avatar hauntsaninja avatar hroncok avatar ilevkivskyi avatar ilinum avatar jaraco avatar michael0x2a avatar mr-c avatar msullivan avatar odidev avatar srittau avatar tacaswell avatar tbbharaj avatar theelx avatar vstinner avatar webknjaz avatar wictory avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

typed_ast's Issues

Segfaults on python 3.4/3.5 with non-ASCII identifiers

Running this program

import typed_ast.ast3
assert typed_ast.ast3.parse("topìc_name").body[0].value.id == "topìc_name"
assert typed_ast.ast3.parse("topìc_name").body[0].value.id == "topìc_name"

consistently dumps core on my Ubuntu 18.04 machine when run with 3.4 and 3.5. 3.6 and 3.7 are fine.

Discovered via zulip/zulip#12139

Does not compile under Cygwin

In a Python 3 inside Cygwin, meaning that it was installed over the Cygwin package manager and is not a Windows one accessed from Cygwin, python3 setup.py build will fail:

running build
running build_py
running build_ext
building '_ast27' extension
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Parser/acceler.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/acceler.o
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Parser/bitset.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/bitset.o
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Parser/grammar.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/grammar.o
ast27/Parser/grammar.c:16:12: warning: ‘Py_DebugFlag’ redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
 extern int Py_DebugFlag;
            ^
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Parser/grammar1.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/grammar1.o
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Parser/node.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/node.o
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Parser/parser.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/parser.o
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Parser/parsetok.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/parsetok.o
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Parser/tokenizer.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/tokenizer.o
ast27/Parser/tokenizer.c:41:7: warning: ‘_Ta27Parser_TokenNames’ redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
 char *_Ta27Parser_TokenNames[] = {
       ^
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Python/ast.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/ast.o
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Python/graminit.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/graminit.o
ast27/Python/graminit.c:2332:9: warning: ‘_Ta27Parser_Grammar’ redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
 grammar _Ta27Parser_Grammar = {
         ^
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Python/mystrtoul.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/mystrtoul.o
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Python/Python-ast.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/Python-ast.o
gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/build=/usr/src/debug/python3-3.4.3-1 -fdebug-prefix-map=/usr/src/ports/python3/python3-3.4.3-1.x86_64/src/Python-3.4.3=/usr/src/debug/python3-3.4.3-1 -Iast27/Include -I/usr/include/python3.4m -c ast27/Custom/typed_ast.c -o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Custom/typed_ast.o
gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/acceler.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/bitset.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/grammar.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/grammar1.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/node.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/parser.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/parsetok.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/tokenizer.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/ast.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/graminit.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/mystrtoul.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/Python-ast.o build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Custom/typed_ast.o -L/usr/lib/python3.4/config -L/usr/lib -lpython3.4m -o build/lib.cygwin-2.6.0-x86_64-3.4/_ast27.cpython-34m.dll
build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/grammar.o: In function `_Py_translatelabels':
/home/phili/typed_ast/ast27/Parser/grammar.c:144: undefined reference to `__imp__Ta27Parser_TokenNames'
/home/phili/typed_ast/ast27/Parser/grammar.c:144:(.text+0x3ac): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp__Ta27Parser_TokenNames'
build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Parser/grammar1.o: In function `Ta27Grammar_LabelRepr':
/home/phili/typed_ast/ast27/Parser/grammar1.c:52: undefined reference to `__imp__Ta27Parser_TokenNames'
/home/phili/typed_ast/ast27/Parser/grammar1.c:52:(.text+0x48): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp__Ta27Parser_TokenNames'
/home/phili/typed_ast/ast27/Parser/grammar1.c:50: undefined reference to `__imp__Ta27Parser_TokenNames'
/home/phili/typed_ast/ast27/Parser/grammar1.c:50:(.text+0xb3): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp__Ta27Parser_TokenNames'
build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/ast.o: In function `compiler_complex_args':
/home/phili/typed_ast/ast27/Python/ast.c:681: undefined reference to `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/ast.c:681:(.text+0x8ec): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `_Py_asdl_seq_new'
build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/ast.o: In function `ast_for_exprlist':
/home/phili/typed_ast/ast27/Python/ast.c:2448: undefined reference to `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/ast.c:2448:(.text+0x117f): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `_Py_asdl_seq_new'
build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/ast.o: In function `ast_for_expr':
/home/phili/typed_ast/ast27/Python/ast.c:1992: undefined reference to `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/ast.c:1992:(.text+0x131b): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/ast.c:2028: undefined reference to `_Py_asdl_int_seq_new'
/home/phili/typed_ast/ast27/Python/ast.c:2028:(.text+0x15ac): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `_Py_asdl_int_seq_new'
/home/phili/typed_ast/ast27/Python/ast.c:2031: undefined reference to `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/ast.c:2031:(.text+0x15cf): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `_Py_asdl_seq_new'
build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/ast.o: In function `ast_for_atom':
/home/phili/typed_ast/ast27/Python/ast.c:1597: undefined reference to `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/ast.c:1597:(.text+0x17dc): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/ast.c:1601: undefined reference to `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/ast.c:1601:(.text+0x17f3): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `_Py_asdl_seq_new'
build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/ast.o: In function `ast_for_trailer':
/home/phili/typed_ast/ast27/Python/ast.c:1817: undefined reference to `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/ast.c:1817:(.text+0x193a): additional relocation overflows omitted from the output
/home/phili/typed_ast/ast27/Python/ast.c:1833: undefined reference to `_Py_asdl_seq_new'
build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/ast.o:/home/phili/typed_ast/ast27/Python/ast.c:1578: more undefined references to `_Py_asdl_seq_new' follow
build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/Python-ast.o: In function `obj2ast_expr':
/home/phili/typed_ast/ast27/Python/Python-ast.c:5516: undefined reference to `_Py_asdl_int_seq_new'
/home/phili/typed_ast/ast27/Python/Python-ast.c:5593: undefined reference to `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/Python-ast.c:5541: undefined reference to `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/Python-ast.c:5618: undefined reference to `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/Python-ast.c:5880: undefined reference to `_Py_asdl_seq_new'
/home/phili/typed_ast/ast27/Python/Python-ast.c:5929: undefined reference to `_Py_asdl_seq_new'
build/temp.cygwin-2.6.0-x86_64-3.4/ast27/Python/Python-ast.o:/home/phili/typed_ast/ast27/Python/Python-ast.c:5284: more undefined references to `_Py_asdl_seq_new' follow
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

/edit: Same for 32 Bit and MSYS2.

Fails to build on pypy 7.0 (stdlib 3.5) as pgenheaders.h doesn't exist in pypy

I'm not sure if this should be filed against typed_ast or pypy, as I can't find many references to pgenheaders.h and no documentation for it. Is this perhaps a situation similar to https://bitbucket.org/pypy/pypy/issues/2687/pypy3-is-missing-pystrhexh-breaking-pysha3 where it's a header file for an internal API?

(pypy3ve) user@host:/home/user/pypy3ve$ pypy3-c --version
Python 3.5.3 (1f86f25937b6ae6c8b25236c35228fac587678bf, Feb 12 2019, 13:00:04)
[PyPy 7.1.0-alpha0 with GCC 7.3.0]
(pypy3ve) user@host:/home/user/pypy3ve$ pypy3-c -m pip install --upgrade typed_ast
Collecting typed_ast
  Using cached https://files.pythonhosted.org/packages/fc/c6/61d6410fc70fda073bd1810f9b7f7022f00146b108f278a0c00041bfe5b0/typed-ast-1.3.1.tar.gz
Building wheels for collected packages: typed-ast
  Building wheel for typed-ast (setup.py) ... error
  Complete output from command /home/user/pypy3ve/bin/pypy3-c -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-usaqh1q6/typed-ast/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-3s26ar_9 --python-tag pp371:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.5
  creating build/lib.linux-x86_64-3.5/typed_ast
  copying typed_ast/conversions.py -> build/lib.linux-x86_64-3.5/typed_ast
  copying typed_ast/ast27.py -> build/lib.linux-x86_64-3.5/typed_ast
  copying typed_ast/__init__.py -> build/lib.linux-x86_64-3.5/typed_ast
  copying typed_ast/ast3.py -> build/lib.linux-x86_64-3.5/typed_ast
  running build_ext
  building '_ast27' extension
  creating build/temp.linux-x86_64-3.5
  creating build/temp.linux-x86_64-3.5/ast27
  creating build/temp.linux-x86_64-3.5/ast27/Parser
  creating build/temp.linux-x86_64-3.5/ast27/Python
  creating build/temp.linux-x86_64-3.5/ast27/Custom
  gcc -pthread -DNDEBUG -O2 -fPIC -Iast27/Include -I/home/user/pypy3ve/include -I/home/user/pypy3/include -c ast27/Parser/acceler.c -o build/temp.linux-x86_64-3.5/ast27/Parser/acceler.o
  ast27/Parser/acceler.c:13:10: fatal error: pgenheaders.h: No such file or directory
   #include "pgenheaders.h"
            ^~~~~~~~~~~~~~~
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for typed-ast
  Running setup.py clean for typed-ast
Failed to build typed-ast
Installing collected packages: typed-ast
  Running setup.py install for typed-ast ... error
    Complete output from command /home/user/pypy3ve/bin/pypy3-c -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-usaqh1q6/typed-ast/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-drrzuuwb/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/pypy3ve/include/site/python3.5/typed-ast:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.5
    creating build/lib.linux-x86_64-3.5/typed_ast
    copying typed_ast/conversions.py -> build/lib.linux-x86_64-3.5/typed_ast
    copying typed_ast/ast27.py -> build/lib.linux-x86_64-3.5/typed_ast
    copying typed_ast/__init__.py -> build/lib.linux-x86_64-3.5/typed_ast
    copying typed_ast/ast3.py -> build/lib.linux-x86_64-3.5/typed_ast
    running build_ext
    building '_ast27' extension
    creating build/temp.linux-x86_64-3.5
    creating build/temp.linux-x86_64-3.5/ast27
    creating build/temp.linux-x86_64-3.5/ast27/Parser
    creating build/temp.linux-x86_64-3.5/ast27/Python
    creating build/temp.linux-x86_64-3.5/ast27/Custom
    gcc -pthread -DNDEBUG -O2 -fPIC -Iast27/Include -I/home/user/pypy3ve/include -I/home/user/pypy3/include -c ast27/Parser/acceler.c -o build/temp.linux-x86_64-3.5/ast27/Parser/acceler.o
    ast27/Parser/acceler.c:13:10: fatal error: pgenheaders.h: No such file or directory
     #include "pgenheaders.h"
              ^~~~~~~~~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/home/user/pypy3ve/bin/pypy3-c -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-usaqh1q6/typed-ast/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-drrzuuwb/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/pypy3ve/include/site/python3.5/typed-ast" failed with error code 1 in /tmp/pip-install-usaqh1q6/typed-ast/

Expose more information about function line numbers

In hopes of solving python/mypy#3871 we probably need to modify typed_ast to expose more information about function line numbers.

If the return type annotation of a function is absent, the FunctionDef node will have returns as None, which means mypy cannot get the line number of where the return type annotation should be. Additionally the actual def line number may be impossible to calculate based on given ast data.

tests needed

Once this is fully integrated with mypy, we can run the mypy tests to check that things are being parsed correctly.

It may be also worth checking if CPython has tests for the AST module we can use.

Base version numbers on Python releases?

Since typed_ast is a fork of asts from specific python versions, I think it would make sense to make the version numbers be tied to the Python release that it was forked from, with a minor version and a patch number appended.

This would make the upcoming release be 3.7.2.0.0 (which by PEP 440 can be abbreviated as 3.7.2).

consider parsing type comments as expressions

Pros:

  • Easier to understand API. (This has already caused some user confusion; see #4.)

Cons:

  • Errors in type comments will become syntax errors. (But misplaced type comments are already syntax errors, so maybe this makes sense.)

Please tag and push 0.6.1 from master

I think the typed-ast 0.6.0 you uploaded to pypi is missing something.

You seem to have tagged commit 6157371, which seems to be a detached head representing Ivan's development branch, not the squashed master. Therefore it does not contain the "has_b" flag on the ast27.Str class which I need to experiment with bytes/str/unicode. Would you mind tagging master with 0.6.1 and pushing a new version to PyPI?

Fails to build with Visual Studio 2010

I was working on appveyor builds of mypy, and when I initially tested Python 3.3 and 3.4, which use the older Visual Studio 2010 compiler, compilation failed. It'd be great if typed_ast were to compile for Python 3.3 and 3.4 on Windows

here is the log:

$ py -3.4 setup.py install
running install
running bdist_egg
running egg_info
writing top-level names to typed_ast.egg-info\top_level.txt
writing typed_ast.egg-info\PKG-INFO
writing dependency_links to typed_ast.egg-info\dependency_links.txt
reading manifest file 'typed_ast.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'typed_ast.egg-info\SOURCES.txt'
installing library code to build\bdist.win32\egg
running install_lib
running build_py
running build_ext
building '_ast27' extension
c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Iast27/Include -IC:\Python34\include -IC:\Python34\include /Tcast27/Parser/acceler.c /Fobuild\temp.win32-3.4\Release\ast27/Parser/acceler.obj
acceler.c
c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Iast27/Include -IC:\Python34\include -IC:\Python34\include /Tcast27/Parser/bitset.c /Fobuild\temp.win32-3.4\Release\ast27/Parser/bitset.obj
bitset.c
c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Iast27/Include -IC:\Python34\include -IC:\Python34\include /Tcast27/Parser/grammar.c /Fobuild\temp.win32-3.4\Release\ast27/Parser/grammar.obj
grammar.c
c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Iast27/Include -IC:\Python34\include -IC:\Python34\include /Tcast27/Parser/grammar1.c /Fobuild\temp.win32-3.4\Release\ast27/Parser/grammar1.obj
grammar1.c
c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Iast27/Include -IC:\Python34\include -IC:\Python34\include /Tcast27/Parser/node.c /Fobuild\temp.win32-3.4\Release\ast27/Parser/node.obj
node.c
c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Iast27/Include -IC:\Python34\include -IC:\Python34\include /Tcast27/Parser/parser.c /Fobuild\temp.win32-3.4\Release\ast27/Parser/parser.obj
parser.c
c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Iast27/Include -IC:\Python34\include -IC:\Python34\include /Tcast27/Parser/parsetok.c /Fobuild\temp.win32-3.4\Release\ast27/Parser/parsetok.obj
parsetok.c
c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Iast27/Include -IC:\Python34\include -IC:\Python34\include /Tcast27/Parser/tokenizer.c /Fobuild\temp.win32-3.4\Release\ast27/Parser/tokenizer.obj
tokenizer.c
ast27/Parser/tokenizer.c(1380) : error C2143: syntax error : missing ';' before 'const'
ast27/Parser/tokenizer.c(1381) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1381) : warning C4047: '=' : 'int' differs in levels of indirection from 'char *'
ast27/Parser/tokenizer.c(1382) : error C2065: 'prefix' : undeclared identifier
ast27/Parser/tokenizer.c(1382) : warning C4047: '=' : 'int' differs in levels of indirection from 'const char *'
ast27/Parser/tokenizer.c(1383) : error C2065: 'prefix' : undeclared identifier
ast27/Parser/tokenizer.c(1383) : error C2100: illegal indirection
ast27/Parser/tokenizer.c(1383) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1383) : warning C4047: '<' : 'int' differs in levels of indirection from 'char *'
ast27/Parser/tokenizer.c(1384) : error C2065: 'prefix' : undeclared identifier
ast27/Parser/tokenizer.c(1384) : error C2100: illegal indirection
ast27/Parser/tokenizer.c(1385) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1385) : error C2100: illegal indirection
ast27/Parser/tokenizer.c(1385) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1385) : error C2100: illegal indirection
ast27/Parser/tokenizer.c(1386) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1387) : error C2065: 'prefix' : undeclared identifier
ast27/Parser/tokenizer.c(1387) : error C2100: illegal indirection
ast27/Parser/tokenizer.c(1387) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1387) : error C2100: illegal indirection
ast27/Parser/tokenizer.c(1388) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1393) : error C2065: 'prefix' : undeclared identifier
ast27/Parser/tokenizer.c(1397) : error C2065: 'prefix' : undeclared identifier
ast27/Parser/tokenizer.c(1397) : error C2100: illegal indirection
ast27/Parser/tokenizer.c(1401) : error C2065: 'type_start' : undeclared identifier
ast27/Parser/tokenizer.c(1401) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1403) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1403) : warning C4047: '>=' : 'char *' differs in levels of indirection from 'int'
ast27/Parser/tokenizer.c(1403) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1403) : warning C4022: 'memcmp' : pointer mismatch for actual parameter 1
ast27/Parser/tokenizer.c(1404) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1405) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1405) : warning C4047: '<' : 'int' differs in levels of indirection from 'char *'
ast27/Parser/tokenizer.c(1406) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1406) : error C2100: illegal indirection
ast27/Parser/tokenizer.c(1408) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1408) : error C2100: illegal indirection
ast27/Parser/tokenizer.c(1408) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1408) : error C2100: illegal indirection
ast27/Parser/tokenizer.c(1409) : error C2065: 'p' : undeclared identifier
ast27/Parser/tokenizer.c(1420) : error C2065: 'type_start' : undeclared identifier

mypy --fast-parser error even when dependency is satisfied?

I have the following Dockerfile:

FROM python:3.6.1

WORKDIR /tmp

RUN apt-get update -y
RUN apt-get install -y git
RUN git clone https://github.com/vim/vim.git && cd vim && ./configure && apt install ncurses-dev && make && make install
RUN vim --version | head -n 1

ADD ./requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt

COPY .vim /root/.vim
COPY .vimrc /root/.vimrc

WORKDIR /app

Here is my requirements file:

flake8==3.2.1
flake8-deprecated==1.1
flake8-docstrings==1.0.3
flake8-mock==0.3
flake8-quotes==0.9.0
mypy==0.470
pep8-naming==0.4.1
pylint==1.6.4
pytest==3.0.5

When I open a Python file inside the container I see the message:

You must install the typed_ast package before you can run mypy with `--fast-parser`.
You can do this with `python3 -m pip install typed-ast`.

But if I install that as suggested I see:

Requirement already satisfied: typed-ast in /usr/local/lib/python3.6/site-packages

await in function with type comment on new line causes SyntaxError

I found this when trying to write tests. :-)

These all work:

async def foo():
    return await bar()

async def foo() -> int:
    return await bar()

async def foo():
    # comment
    return await bar()

async def foo(): # type: () -> int
    return await bar()

async def foo(): # type: () -> int
    return 0

But this gives a syntax error on the await:

async def foo():
    # type: () -> int
    return await bar()

No type comments on AsyncWith/AsyncFor?

From Python.asdl in ast35:

For(expr target, expr iter, stmt* body, stmt* orelse, string? type_comment)
With(withitem* items, stmt* body, string? type_comment)
but
AsyncFor(expr target, expr iter, stmt* body, stmt* orelse)
AsyncWith(withitem* items, stmt* body)

I don't believe there's anything against using type comments in async withs/fors, and this limits how MyPy can analyze them.

Change names of ast* modules

Instead of adding a new ast3x module for every new Python version, we're now planning on updating the ast35 module with the Python changes. Accordingly, the ast27 and ast35 modules should probably be renamed to ast2 and ast3, respectively.

Issue with string length using Latin-1 encoding

Hello.

I'm working on a bug affecting pylint, which uses astroid, which uses typed_ast:
pylint-dev/pylint#2610

What troubles me is the .col_offset value of a Str node containing a non-ASCII character with Latin1 encoding.

Here is a minimal code example:

# coding: latin_1
from typed_ast import ast3
print(ast3.parse("'a'+'A'").body[0].value.right.col_offset)  # print: 4
print(ast3.parse("'à'+'A'").body[0].value.right.col_offset)  # print: 5

From my understanding a "é" character as a length of 1 in Latin-1:
https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Code_page_layout

Hence, could someone help me understanding why the .col_offset is higher in that case, please ?

SyntaxError when parsing per-argument type comments with comma on the next line

In the PR adding per-argument type comment support #5 there is an unhandled case when comma for an argument is after the type comment, therefore the current version of typed_ast errors out on the following.

In [1]: import inspect

In [2]: import typed_ast.ast3

In [3]: def lovely(spam  # type: bool
   ...:         , eggs  # type: int
   ...:         , ham  # type: str
   ...:         ):
   ...:     # type: (...) -> bool
   ...:     return spam
   ...: 

In [4]: typed_ast.ast3.parse(inspect.getsource(lovely))
  File "<unknown>", line 2
    , eggs  # type: int
    ^
SyntaxError: invalid syntax

Are there any plans to support this case?

In [5]: import ast

In [6]: ast.parse(inspect.getsource(lovely))
Out[6]: <_ast.Module at 0x7fb9584b84e0>

The built-in ast succeeds, which seems to be in conflict with the development philosophy of typed_ast.

This project is a drop-in replacement for the builtin ast module. It is intended to be bug-for-bug compatible and behave identically, except for the presence of a few additional fields on the returned classes and a few additional optional arguments to the parse call.

Add ast36

I've just tried mypy --fast-parser --python-version 3.6 on code that utilizes some new Python 3.6 features and it failed.

Breaking features:

  • formatted string literals, PEP 498 -- error: invalid syntax
  • asynchronous generators, PEP 525 -- error: 'yield' in async function
  • asynchronous comprehensions, PEP 530 -- error: invalid syntax

parsing f-strings -- opening brace gets duplicated when backslash is followed by an expression

with Python 3.6.0 and typed-ast 1.0.2, I get the following:

#!/usr/bin/env python3.6

import typed_ast.ast3 as ast

code1 = '''"\\{x}"'''
code2 = '''f"\\{x}"'''

tree1 = ast.parse(code1, mode='eval')
print(ast.dump(tree1))
tree2 = ast.parse(code2, mode='eval')
print(ast.dump(tree2))

output:

Expression(body=Str(s='\\{x}'))
Expression(body=JoinedStr(values=[Str(s='\\{'), FormattedValue(value=Name(id='x', ctx=Load()), conversion=-1, format_spec=None)]))

Therefore, the normal string is '\\{x}'.
But the f-string has two parts: '\\{' and an expression Name(id='x', ctx=Load()).

Where does the { in the string part of f-string come from? This happens also with built-in ast. I can't believe this is the intended behavior... Is it?

Misplaced type annotations should be a warning or list

Given the Python code:

def wrong1():
    "A docstring."
    # type: () -> int
    return 42


def wrong2():
    "A docstring."
    # type: () -> int
    return 42

typed_ast raises a SyntaxError on the first misplaced comment:

In [8]: ast27.parse("""def wrong1():
   ...:     "A docstring."
   ...:     # type: () -> int
   ...:     return 42
   ...:
   ...:
   ...: def wrong2():
   ...:     "A docstring."
   ...:     # type: () -> int
   ...:     return 42
   ...: """)
Traceback (most recent call last):

  File "/users/wilfred/.conda/envs/ast_scratch/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-ebf02dacb9b0>", line 11, in <module>
    """)

  File "/users/wilfred/.conda/envs/ast_scratch/lib/python3.6/site-packages/typed_ast/ast27.py", line 51, in parse
    return _ast27.parse(source, filename, mode)

  File "<unknown>", line 3
    # type: () -> int
                    ^
SyntaxError: misplaced type annotation

This means that it's not possible to get a list of all the misplaced type annotations.

Are there any specs for the typed AST?

I've started experimenting with this project and it works really nice so far, but what I find that is missing is the description of the typed AST. For ordinary AST there is some effort going on here: https://greentreesnakes.readthedocs.io/en/latest/nodes.html . However, is there any kind of documentation with regard to how typed_ast.ast35 should look?

E.g. type_comment seems to be treated as just a string. However what is done e.g. in mypy is the type comment is treated as code. Is that intended?

my_string = None # type: typing.Optional[str]
Module(
  body=[Assign(
    targets=[Name(
      id='my_string',
      ctx=Store())],
    value=NameConstant(value=None),
    type_comment='typing.Optional[str]')],
  type_ignores=[])

Another example is here, where type annotations are not treated as strings, but are parsed normally:

def function(a: int, b: int) -> int:
    return a + b
Module(
  body=[FunctionDef(
    name='function',
    args=arguments(
      args=[
        arg(
          arg='a',
          annotation=Name(
            id='int',
            ctx=Load())),
        arg(
          arg='b',
          annotation=Name(
            id='int',
            ctx=Load()))],
      vararg=None,
      kwonlyargs=[],
      kw_defaults=[],
      kwarg=None,
      defaults=[]),
    body=[Return(value=BinOp(
      left=Name(
        id='a',
        ctx=Load()),
      op=Add(),
      right=Name(
        id='b',
        ctx=Load())))],
    decorator_list=[],
    returns=Name(
      id='int',
      ctx=Load()),
    type_comment=None)],
  type_ignores=[])

Copy f-string line/col fix from CPython

There's an issue with f-string line/column numbers which was reported for mypy but is actually an ast bug that also exists upstream. (We don't want to get in the business of fixing those independently from upstream.)

It's been fixed upstream in python/cpython#1800 (bpo-30465) so the fix should be incorporated here so we can eventually declare victory in mypy.

Segfault on f-string in ast3

This happens:

Python 3.5.1 |Anaconda custom (x86_64)| (default, Dec  7 2015, 11:24:55) 
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin

>>> from typed_ast import ast3
>>> ast3.parse("x = f'Hi {x}'")
Segmentation fault: 11

Unfortunately, I could not test this on other versions of Python now.

mypy --fast-parser option broken with 1.0.0

Running in a Python 3.6.0 virtual environment, running mypy with --fast-parser option does not work with typed-ast==1.0.0:

$ pip freeze | grep 'mypy\|typed-ast'
mypy-lang==0.4.6
typed-ast==1.0.0

$ mypy --fast-parser .
You must install the typed_ast package before you can run mypy with `--fast-parser`.
You can do this with `python3 -m pip install typed-ast`.

The same run works with typed-ast==0.6.3

Inconsistent parsing of f-strings between typed_ast and the builtin ast

Hey folks!

I've noticed that typed_ast doesn't parse f-strings in the same way as the builtin ast module does. For instance in the following example, we only get a FormattedValue while the builtin ast module returns a JoinedStr:

import typed_ast.ast3 as typed_ast
import ast

f = typed_ast.parse("f'{a}'")
print(f.body[0].value)

f = ast.parse("f'{a}'")
print(f.body[0].value)

# going to print the following
<_ast3.FormattedValue object at 0x1040a09e8>
<_ast.JoinedStr object at 0x1040a0b00>

It would be great to have the same output as the builtin ast module. They both return a JoinedStr if the variable interpolation element is followed by one or more characters.

Str.kind exists but Bytes.kind doesn't - what's the design here?

I'm not sure if it's a bug, or a feature, but at least to me the following:

- `Str` has a `kind` string field which preserves the original string
 prefix, so that `ast3.parse('u"test"').body[0].value.kind == 'u'`. 

from https://github.com/python/typed_ast/blob/master/typed_ast/ast3.py#L24 suggests that raw strings are also supported, and indeed they are. However, in case of raw bytes the prefix is lost. Is this by design?

Currently it's like this:

In [1]: import typed_ast                                                                                 

In [2]: typed_ast.__version__                                                                            
Out[2]: '1.2.0'

In [3]: from typed_ast.ast3 import parse, dump                                                           

In [4]: dump(parse('''r"spam"'''))                                                                       
Out[4]: "Module(body=[Expr(value=Str(s='spam', kind='r'))], type_ignores=[])"

In [5]: dump(parse('''rb"spam"'''))                                                                      
Out[5]: "Module(body=[Expr(value=Bytes(s=b'spam'))], type_ignores=[])"

Preserving the original prefix of string is useful for many reasons, same can be said about preserving the original bytes prefix, i.e. parsing rb"spam" into for example Bytes(s=b'spam', kind='rb').

However, as I'm not sure what was the reason behind adding that kind field to Str, I'm not sure if the same reason can be applied to Bytes as well. And by the way, is there any place where the choice of preserving kind at all is explained?

Error when trying to install with Python 2 pip is really bad

I accidentally ran pip install typed_ast using Python 2 and got these errors:

[many more errors from gcc skipped]

ast27/Parser/tokenizer.c:512:45: error: ‘PyId_readline’ undeclared (first use in this function)

     readline = _PyObject_GetAttrId(stream, &PyId_readline);

                                             ^

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-uAt4J4/typed-ast/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-oRixWl-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip-build-uAt4J4/typed-ast
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    load_entry_point('pip==1.5.6', 'console_scripts', 'pip')()
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 248, in main
    return command.main(cmd_args)
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 161, in main
    text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 41: ordinal not in range(128)

Surely the terrible final error message is not typed_ast's fault, but can the package just be marked as Python 3-only on PyPI somehow?

Misplaced type annotation when "type:" appears anywhere in a comment

I originally opened pylint-dev/pylint#2299 but it looks like this belongs here. I'm not sure how much of this is actually intended (or in some PEP), vs. what's actually how things are implemented in typed-ast, though.

I have some comments like those which raise a syntax error:

# The Mozilla Developer Network says:
# type: This attribute defines the styling language as a MIME type
# (charset should not be specified). This attribute is optional and
# default to text/css if it's missing.
# https://developer.mozilla.org/en/docs/Web/HTML/Element/style
# At least in init(), we can get errors like this:
# type: ConnectionError
# database text: out of memory
# driver text: Error opening database
# error code: -1

i.e. type: just happens to appear in those comments without actually being type comments. Unfortunately, even indenting those lines (# type: ConnectionError) or adding some other text before it (# > type: ConnectionError) still makes it fail.

Not sure what the best solution here is, though - it seems to me that at least, only # type: ... should actually name a type, but not # some other stuff type: ..., no?

Support end line and end column?

Recently support for end line and end column of AST nodes was merged in Python 3.8. I think it would be great to support it in typed_ast (to have it on older Python versions too). If we agree this makes sense, then I can make a PR.

tag releases so they're visible on GitHub

You should add git tags to the release commits (e.g. v0.5.5) so that it's easy to see what the releases have been. (might also be worth adding a changelog, but that's more work).

Support 3.7

Currently python 3.7 is unsupported. Is it to early to introduce support right now? @ethanhs seems to be making progress, not sure if it's stable.

Parse error for Long octal literals in ast27

01L is a perfectly valid Python 2 long literal, but for some reason ast27 refuses to parse it and instead gives ValueError: invalid literal for int() with base 0: '01'. 01 and 1L both work fine.

Error installing pylint for python3.7 on cygwin

Hello,

I had an issue that looked almost the same as #63 when trying to install pylint for python3.7 on cygwin.

Here is the issue I originally opened to pylint : pylint-dev/pylint#2886

As additional information :
pjtallon@------- ~
$ which gcc
/usr/bin/gcc

pjtallon@------- ~
$ gcc --version
gcc (GCC) 7.4.0

Best,
Julien

Please provide a changelog

And include it in the published tarballs.

As packager I can then pass them on to the users in the packages.

Build failure against Python 3.6 (adddfa function signature conflict)

v1.3.1 fails build for Python 3.6 using gcc 8.2.0:

pushd /var/tmp/paludis/build/dev-python-typed-ast-1.3.1/work/PYTHON_ABIS/3.6/typed-ast-1.3.1
python3.6 -B setup.py build
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/typed_ast
copying typed_ast/ast27.py -> build/lib.linux-x86_64-3.6/typed_ast
copying typed_ast/ast3.py -> build/lib.linux-x86_64-3.6/typed_ast
copying typed_ast/__init__.py -> build/lib.linux-x86_64-3.6/typed_ast
copying typed_ast/conversions.py -> build/lib.linux-x86_64-3.6/typed_ast
running build_ext
warning: build_py: byte-compiling is disabled, skipping.

building '_ast27' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/ast27
creating build/temp.linux-x86_64-3.6/ast27/Parser
creating build/temp.linux-x86_64-3.6/ast27/Python
creating build/temp.linux-x86_64-3.6/ast27/Custom
x86_64-pc-linux-gnu-cc -DNDEBUG -fwrapv -I/usr/x86_64-pc-linux-gnu/include/python3.6m -pipe -march=native -mtune=native -O2 -fdiagnostics-color=always -fPIC -Iast27/Include -I/usr/include/python3.6m -c ast27/Parser/acceler.c -o build/temp.linux-x86_64-3.6/ast27/Parser/acceler.o
ast27/Parser/acceler.c: In function 'fixstate':
ast27/Parser/acceler.c:88:23: warning: implicit declaration of function 'Ta27Grammar_FindDFA'; did you mean 'PyGrammar_FindDFA'? [-Wimplicit-function-declaration]
             dfa *d1 = Ta27Grammar_FindDFA(g, type);
                       ^~~~~~~~~~~~~~~~~~~
                       PyGrammar_FindDFA
ast27/Parser/acceler.c:88:23: warning: initialization of 'dfa *' {aka 'struct <anonymous> *'} from 'int' makes pointer from integer without a cast [-Wint-conversion]
x86_64-pc-linux-gnu-cc -DNDEBUG -fwrapv -I/usr/x86_64-pc-linux-gnu/include/python3.6m -pipe -march=native -mtune=native -O2 -fdiagnostics-color=always -fPIC -Iast27/Include -I/usr/include/python3.6m -c ast27/Parser/bitset.c -o build/temp.linux-x86_64-3.6/ast27/Parser/bitset.o
x86_64-pc-linux-gnu-cc -DNDEBUG -fwrapv -I/usr/x86_64-pc-linux-gnu/include/python3.6m -pipe -march=native -mtune=native -O2 -fdiagnostics-color=always -fPIC -Iast27/Include -I/usr/include/python3.6m -c ast27/Parser/grammar.c -o build/temp.linux-x86_64-3.6/ast27/Parser/grammar.o
In file included from ast27/Parser/grammar.c:5:
/usr/x86_64-pc-linux-gnu/include/python3.6m/pgenheaders.h:19:16: error: conflicting types for '_Py_adddfa'
 #define adddfa _Py_adddfa
                ^~~~~~~~~~
ast27/Parser/grammar.c:36:1: note: in expansion of macro 'adddfa'
 adddfa(grammar *g, int type, char *name)
 ^~~~~~
/usr/x86_64-pc-linux-gnu/include/python3.6m/pgenheaders.h:19:16: note: previous declaration of '_Py_adddfa' was here
 #define adddfa _Py_adddfa
                ^~~~~~~~~~
/usr/x86_64-pc-linux-gnu/include/python3.6m/grammar.h:73:6: note: in expansion of macro 'adddfa'
 dfa *adddfa(grammar *g, int type, const char *name);
      ^~~~~~

I don't see pgenheaders.h anywhere in source folder. I expect that it should be gneerated with entries like:

 #define adddfa _Py27_adddfa  /* ast27 */

pgen instructions in readme

I followed the instructions to get a working pgen binary and ran into the following compilation error:

Parser/grammar1.c: In function ‘PyGrammar_LabelRepr’:
Parser/grammar1.c:50:20: error: ‘_PyParser_TokenNames’ undeclared (first use in this function)
             return _PyParser_TokenNames[lb->lb_type];
                    ^
Parser/grammar1.c:50:20: note: each undeclared identifier is reported only once for each function it appears in
Parser/grammar1.c:61:1: warning: control reaches end of non-void function [-Wreturn-type]
 }

It is mentioned in the README that we need to overwrite Include/token.h, Include/compile.h, and Parser/tokenizer.c, I assume that's referring to replacing those in cpython with those in typed_ast/ast35 right?

typed-ast 1.1 adds new gcc requirement

We use typed-ast indirectly via pylint -> asteroid -> typed-ast for lightweight CI tests. They run in a super minimal Alpine Linux setup to keep it fast and light. Up until typed-ast 1.1, it worked fine without gcc installed. With 1.1, it says gcc is required. It would be really useful to not enforce that again, like in 1.0.4.

https://gitlab.com/eighthave/fdroidserver/-/jobs/82274815

Installing collected packages: pycodestyle, pyflakes, mccabe, six, typed-ast, wrapt, lazy-object-proxy, astroid, isort, pylint, Click, chardet, certifi, urllib3, idna, requests, pyparsing, packaging, pyyaml, dparse, safety
  Running setup.py install for typed-ast: started
    Running setup.py install for typed-ast: finished with status 'error'
    Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-du7x283w/typed-ast/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-kpm5idph-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/typed_ast
    copying typed_ast/ast3.py -> build/lib.linux-x86_64-3.6/typed_ast
    copying typed_ast/ast27.py -> build/lib.linux-x86_64-3.6/typed_ast
    copying typed_ast/__init__.py -> build/lib.linux-x86_64-3.6/typed_ast
    copying typed_ast/conversions.py -> build/lib.linux-x86_64-3.6/typed_ast
    running build_ext
    building '_ast27' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/ast27
    creating build/temp.linux-x86_64-3.6/ast27/Parser
    creating build/temp.linux-x86_64-3.6/ast27/Python
    creating build/temp.linux-x86_64-3.6/ast27/Custom
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -Iast27/Include -I/usr/include/python3.6m -c ast27/Parser/acceler.c -o build/temp.linux-x86_64-3.6/ast27/Parser/acceler.o
    unable to execute 'gcc': No such file or directory
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-du7x283w/typed-ast/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-kpm5idph-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-du7x283w/typed-ast/
You are using pip version 9.0.1, however version 10.0.1 is available.

Uninitialized memory read related to type_comment for *args or **kwds

I believe I've found a very shy error due to an uninitialized memory read in typed_ast.

The repro conditions are murky, it only works on a particular rev of a particular codebase. The problem manifests itself in one of two ways:

  • SystemError: <built-in function _parse> returned a result with an error set with a traceback ending at line 54 in ast3.py; I suspect it is getting the type_comment from a struct that doesn't have that field and reading nonsense. (Most of the time the memory was freshly allocated and the nonsense happens to look like a NULL pointer which then gets treated as intended, but apparently in some scenarios it returns non-NULL garbage.)

  • Function has duplicate type signatures from make_argument() in transform_args() in mypy/fastparse.py; this for a line that is actually correct; so far all examples I've seen involve *args and I suspect the true cause is a similar scenario as the first bullet, a non-NULL piece of garbage.

ISTR we fixed a similar issue in the past.

I will investigate more next week.

Version not matching - typed-ast

Describe the bug

I installed typed-ast 1.3.1 a few days without any issue, from today when installing it says typed-ast version not matching like following screenshot.
image

Expected behavior

pipenv install should be installed correctly without issue.

To Reproduce

pipenv install typed-ast
My Platform

Ubuntu 18.04
Python3.6.7
Pi 19.0.3
SSL/TLS implemented

Additional context

My Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[dev-packages]
"autopep8" = "*"
"gprof2dot" = "*"
"pep8" = "*"
"pyprof2calltree" = "*"
"urllib3" = "*"
asynctest = "*"
autoflake = "*"
awscli = "*"
certifi = "*"
crayons = "*"
cython = "*"
ipython = "*"
isort = "*"
jsonschema = "*"
jupyter = "*"
line-profiler = "*"
mypy = "*"
pre-commit = "*"
progress = "*"
pydevd = "*"
pylint = "*"
pytest = "==3.10.0"
pytest-asyncio = "==0.9.0"
pytest-console-scripts = "==0.1.7"
pytest-cov = "==2.6.0"
pytest-docker = "==0.6.1"
pytest-mock = "==1.10.0"
pytest-profiling = "*"
pytest-pylint = "*"
pytest-sanic = "*"
pytest-timeout = "*"
python-rapidjson = "*"
requests = "*"
yapf = "*"
funcy = "*"
vprof = "*"

[packages]
aiodns = "*"
aiohttp = "*"
aredis = "*"
cchardet = "*"
configargparse = "*"
cytoolz = "*"
hiredis = "*"
pipenv = "*"
pygtrie = "*"
python-json-logger = "*"
python-rapidjson = "*"
sanic = "*"
structlog = "*"
ujson = "*"
uvloop = "*"
websockets = "*"
async-timeout = "*"
typed-ast = "*"

[requires]
python_version = "3.6"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.