Many of the codemods I write would benefit from knowing whether a given file is compatible with python 2, python 3, or both. May as well report minor versions while we're at it. There's a long tail of things, but looking in the tree for print statements (2), f-strings (3) and typehints (3) are probably 90% of the way there.

Wherever this goes, we should be able to check its result from a filename_matcher as well as from a modifier func.

This bowler script fails with "function spec invalid" for source files that import the name:

Query(...).select_function("getenv").add_argument("b").diff()

e.g. on

from os import getenv
getenv("a")

Query.select_method doesn't have this problem, only select_function.

Query().write() prompts for every hunk, rather than assuming yes for each hunk.

Repro

git clone https://github.com/jreese/aql
cd aql
git checkout 3565e740
bowler do  'Query().select_class("Operation").rename("Comparison").write()'

Expected result

Bowler should just apply all modified hunks without prompting (and without printing hunks, IMO).

Actual result

Bowler prompts the user to apply each hunk, essentially identical to idiff() or diff(interactive=True).

Input file:

from commonmark.main import commonmark

Code:

bowler.Query('__init__.py').select_module('commonmark').rename('testme').execute(write=True, silent=True) 

Output:

from testme.main import testme

docs/api-query [@548fc816fab01c933bf06618670dddfeb43e9c09] references a select_name function, but I can't find by grepping the source code.

If you use .filter("power < any* >") instead of .select, the resulting error is confusing because it just eval's string arguments. What's the usecase for .filter taking a string? I'd like to improve the error message if < is found in it as a first effort.

For example, you want to rename node only if it matches some pattern. You can do this:

def modify(node: LN, capture: Capture, filename: Filename) -> None:
  if re.match(...):
    rename_transform(...)

However, rename_transform isn't available for import. BTW, it's can be really useful to have bowler.modifiers with some common modifiers to reuse them. Maybe, moving rename_transform in it is a good start. Same for add_argument_transform, encapsulate_transform etc

And what if use _modifier instead of _transform?

I can do PR for it if you agree with the point.

Hi,

First of all thanks for this great project. It's helping on modernizing the architecture of GraphQL Python projects.

Use Case

I'm trying to port automatically Python 3 code to Python 2 (to not break compatibility for old projects using the library, similar to what babel is doing).
On this quest, I was trying to replace the calls to my_var = cast(MyType, old_var) to my_var = old_var.

For it, I created the following pattern and modifier:

example.py

# This is the file that we want to modify
my_var = cast(MyType, old_var)

bowler_file

from bowler import Query

query = Query("example.py")

cast_pattern = """
    power< "cast"
        trailer< "(" cast_args=any* ")" >
    >
"""

def cast_modifier(node, capture, filename):
    args = capture.get("cast_args")
    last_arg = args[-1].children[-1]
    return last_arg

query.select(cast_pattern).modify(cast_modifier).diff()

The modifier is returning a different node than the provided as argument in the modifier function, so it should be replaced.

However, based on the current implementation the return value of the fixer & callback is actually not used, therefore no matter what the fixer or callback returns... the return value will always be skipped, forcing to have always the same node on the modifiers.

It seems this specific case is documented, however is not implemented.

Returned values will automatically replace the matched element in the syntax tree.
https://pybowler.io/docs/api-query#modify

Use a more specific return type at https://github.com/facebookincubator/Bowler/blob/master/bowler/types.py#L47

Right now it's Any, and seems like it should be Optional[LN]

1. Prepare a change to do that (see getting started docs)
2. Verify that mypy is happy with it using make lint
3. Send a PR!

As a developer in charge of CI and automation I want this project distributed to pypi as a wheel because wheels are faster to deploy, and they don't require the runtime version of python to be available when running pip install. The vast majority of projects are using wheels now, and it's really not much more work for the maintainer. Another thing that should be fixed is that the project should be marked as python3-only on PyPI.

thanks!

There's a bug on https://github.com/facebookincubator/Bowler/blob/master/bowler/tool.py#L214 when used with refactor_dir; it takes the count of args as number of processes. This doesn't hit the case of bowler run foo.py $(find . -name '*.py') which is how I normally do it. Removing the min is the easiest way forward.

It seems that select_function doesn't match function calls that are preceeded by await. In the example below a synchronous file works as expected, but the async example only changes the name of the function at the definition, not the call.

Is there a different way to select and modify async function calls?

sync.py

def first_func():
    print("In first func")

def main():
    first_func()

if __name__ == '__main__':
    main()

async.py

import asyncio

async def first_func():
    print("In first func")

async def async_main():
    await first_func()

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(async_main())

bowler command

$ bowler do "Query(['sync.py', 'async.py']).select_function('first_func').rename('old_first_func').diff()"
--- async.py
+++ async.py
@@ -1,6 +1,6 @@
 import asyncio

-async def first_func():
+async def old_first_func():
     print("In first func")


--- sync.py
+++ sync.py
@@ -1,10 +1,10 @@

-def first_func():
+def old_first_func():
     print("In first func")


 def main():
-    first_func()
+    old_first_func()


 if __name__ == '__main__':

I wrote a thing to convert self.assertFalse(x) to assert not x, etc. Trouble is, it falls over easily when the expression in the parentheses isn't valid without the parentheses:

self.assertFalse(
    'a'
    'b'
)

My code naively converts this to

assert not 'a'
    'b'

• I think a parenthesize_if_necessary helper would be really useful for this kind of situation. I imagine this bug is widespread. I tried to write one, but I'm a bit out of my depth.
• It'd be useful to have access to a function which asserts a list of nodes has no syntax errors. I realise that would vary depending on targeted python version and __future__ imports, but perhaps something can be done?

Thanks

Is it possible to change all call args to kwargs?!? Any maybe change all signatures to e.g.: (*, foo=None)

before e.g.:

class Foo:
    def __init__(self, foo, bar):
        self.foobar(foo, bar)

    def foobar(self, foo, bar):
        print(foo, bar)

Foo("one", "two")

after:

class Foo:
    def __init__(self, *, foo, bar):
        self.foobar(foo=foo, bar=bar)

    def foobar(self, *, foo, bar):
        print(foo, bar)

Foo(foo="one", bar="two")

PR's and commits aren't triggering builds on Travis. We should fix that.

Input file:

import attr


@attr.s()
class A:
  ...

Code:

import bowler
bowler.Query('tmp.py').select_module('attr').rename('testme').execute(write=True, silent=True)

Output file:

import testme


@attr.s()
class A:
  ...

Attr was renamed in the import statement, but not in the decorator.

Currently, Bowler does not support renames of modules that involve changing their parent import path. Eg, "some.path" -> "some.path2" is easy with .rename(), but "some.long.path" -> "shorter.path" isn't yet supported. This shouldn't be too difficult, just needs a special case in the existing .rename() modifier to handle this for module imports.

• isort wants to change imports on a fresh checkout (should we do another run of black after?)
• mypy requires a python3.6 executable, Arch only has 3.7 by default

Unrelated to the sh problem in windows I've caught another one:

Traceback (most recent call last):
  File "E:\packages\try_bowler\test.py", line 7, in <module>
    rename.dump()
  File "C:\Users\Levitanus\AppData\Local\Programs\Python\Python37-32\lib\site-packages\bowler\query.py", line 967, in dump
    return self.execute(write=False)
  File "C:\Users\Levitanus\AppData\Local\Programs\Python\Python37-32\lib\site-packages\bowler\query.py", line 957, in execute
    self.retcode = BowlerTool(fixers, **kwargs).run(self.paths)
  File "C:\Users\Levitanus\AppData\Local\Programs\Python\Python37-32\lib\site-packages\bowler\tool.py", line 342, in run
    self.refactor(paths)
  File "C:\Users\Levitanus\AppData\Local\Programs\Python\Python37-32\lib\site-packages\bowler\tool.py", line 223, in refactor
    child.start()
  File "C:\Users\Levitanus\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Users\Levitanus\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Levitanus\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\Levitanus\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\Levitanus\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Query.create_fixer.<locals>.Fixer'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Levitanus\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\spawn.py", line 99, in spawn_main
    new_handle = reduction.steal_handle(parent_pid, pipe_handle)
  File "C:\Users\Levitanus\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\reduction.py", line 87, in steal_handle
    _winapi.DUPLICATE_SAME_ACCESS | _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Отказано в доступе
[Finished in 0.3s with exit code 1]

Original import in tool.py was replaced to import pbs as sh, but. as I understant, flow hasn't gone to the patching part of refactor

original source:

import bowler as bl


rename = bl.Query()
rename.select_var('value')
rename.rename('new_val')
rename.dump()

I apologise in advance if this is invalid or a beginner mistake.
Using query.select_subclass('something.Foo') in following definition does not yield results:

class Bar(something.Foo):
    pass

Workaround is to use select with pattern:

pattern = """
    class_def=classdef<
        'class' class_name=any '('
        (
            power<
                name="something"
                trailer<
                    any*
                    name="Foo"
                >
            >
        )
    >
"""

This appears to be valid in 3.6, 3.7, and 3.8.0a2:

def func(**kwargs):
    print(kwargs)

x = {'a': '1'}
func(**x or {})

Bowler can't parse it, I'm wondering if there is a fissix issue? We should also show the error near "except ParseError" in bowler/tool.py.

Hi.
I have some code that uses format string literals and .format() like below:

def foo(bar, zar):
    print(f'bar is {bar}, not {zar}')
    print('bar is {bar}, not {zar}'.format(bar=bar, zar=zar))

I want to rename the variable name bar to bar_new so the rename.py script is ready:

from bowler import Query

Query('.').select_var('bar').rename('bar_new').idiff()

I hope Bowler recognize the placeholder in string literal, {bar}, but it wasn't:

--- ./format_string.py
+++ ./format_string.py
@@ -1,3 +1,3 @@
-def foo(bar, zar):
+def foo(bar_new, zar):
   print(f'bar is {bar}, not {zar}')
-  print('bar is {bar}, not {zar}'.format(bar=bar, zar=zar))
+  print('bar is {bar}, not {zar}'.format(bar_new=bar_new, zar=zar))
Apply this hunk [y,N,q,a,d,?]? 

I want Bowler to rename

• f'bar is {bar}, not {zar}' → f'bar is {bar_new}, not {zar}'
• .format(bar=bar, zar=zar) → '.format(bar=bar_new, zar=zar)

and this is what exactly the Refactor → Rename function does in PyCharm.

Is there a way to refactor my code like this?
Thank you for your great tool.

I am trying to perform a select like:

.select(r"funcdef< any* ':' suite< '\n' firstnode='    ' any* > >")

This indent which I have labelled firstnode will have the mypy Python 2 type annotation comment attached to it as .prefix if the funcdef has one

My problem is that for more deeply nested functions the indent string I need to match may be 8 spaces, or 12.

It seems like it'd be really nice to be able to use an abstract INDENT token in the pattern rather than the specific string of space chars.

I am assuming there is nothing in the pattern grammar which will help here? I assume if I do like ' '+ (four spaces, repeated - not even sure if that is valid) it won't match ' ' (eight spaces)?

This could provide a nice entry point for discovering and/or executing BowlerTestCases contained within codemod files.

Now selectors captures everything related to the given topic. For example, select_module captures module_name, module_nickname etc. Sometimes it's useful to be more specific. For example:

...select_module('test', capture=['module_name'])

For example, this snippet with a null transform will fail validation. The current (even as of 3.8a) lib2to3 can parse it, but recent versions of ast will fail.

try:
  pass
except Exception, e:
  pass

I've been kicking around the idea of reparsing with lib2to3 and doing a tree compare as part of testing, but maybe a less strict version is just do the parse without compare...

For a recent project I wished there were some functions that would let me figure out int vs float (vs unknown) for simple expressions. I'm not aiming for this to be a full type-checker, or even go beyond those three (int/float/unknown), but I'll run the experiment to see if this ends up being too many lines of code.

Works (output):

$ echo 'pass' > foo
$ bowler dump foo
foo
[file_input] ""
.  [simple_stmt] ""
.  .  [1] "" "pass"
.  .  [4] "" "
"
.  [0] "" ""

Doesn't work (no output):

$ echo -n 'pass' > foo
$ bowler dump foo

This seems to apply to the whole file (not finishing in a newline means no output at all for the whole file.)

To better support Windows (#8) and improve reliability on different environments, use of the sh module for the patch binary should be replaced with use of diff-match-patch or a similar module.

Right now it looks like any exception causes a hang.

As a developer in charge of CI and automation I want this project distributed to pypi as a wheel because wheels are faster to deploy, and they don't require the runtime version of python to be available when running pip install. The vast majority of projects are using wheels now, and it's really not much more work for the maintainer. Another thing that should be fixed is that the project should be marked as python3-only on PyPI.

thanks!

Work with bowler quite easy. However, making custom modifier can be tricky. Now there is only one example of it. Do you have more of them? Developers love examples :)

Unfortunately facebook bowler does not return this project, nor is it easy to do a search for bowler + problem or facebook bowler + problem to see if anyone else is having this issue.

Cutesy names are cool and all, but there is so much noise looking for bowler that it makes it hard to use the project.

Currently it doesn't appear possible to set a breakpoint with import pdb; pdb.set_trace() in a BowlerTestCase. It looks like in_process is set to true when calling the modifier, but I still get a traceback when I attempt to use a debugger in the modifier.

I think it could be helpful to support using pdb in test cases.

from bowler.tests.lib import BowlerTestCase
from bowler.types import Leaf, TOKEN


class ExampleTestCase(BowlerTestCase):
    def test_modifier_return_value(self):
        input = "a+b"

        def modifier(node, capture, filename):
            new_op = Leaf(TOKEN.MINUS, "-")
            import pdb; pdb.set_trace() # works without this statement
            return new_op

        output = self.run_bowler_modifier(input, "'+'", modifier)
        self.assertEqual("a-b", output)

self = <refactoring.test.ExampleTestCase testMethod=test_modifier_return_value>

    def test_modifier_return_value(self):
        input = "a+b"
    
        def modifier(node, capture, filename):
            new_op = Leaf(TOKEN.MINUS, "-")
            import pdb; pdb.set_trace()
            return new_op
    
>       output = self.run_bowler_modifier(input, "'+'", modifier)

test.py:17: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <refactoring.test.ExampleTestCase testMethod=test_modifier_return_value>, input_text = 'a+b', selector = "'+'"
modifier = <function ExampleTestCase.test_modifier_return_value.<locals>.modifier at 0x11fc7c598>, selector_func = None, modifier_func = None, in_process = True
query_func = <function BowlerTestCase.run_bowler_modifier.<locals>.default_query_func at 0x11fcaaf28>

    def run_bowler_modifier(
        self,
        input_text,
        selector=None,
        modifier=None,
        selector_func=None,
        modifier_func=None,
        in_process=True,
        query_func=None,
    ):
        """Returns the modified text."""
    
        if not (selector or selector_func or query_func):
            raise ValueError("Pass selector")
        if not (modifier or modifier_func or query_func):
            raise ValueError("Pass modifier")
    
        exception_queue = multiprocessing.Queue()
    
        def store_exceptions_on(func):
            @functools.wraps(func)
            def inner(node, capture, filename):
                # When in_process=False, this runs in another process.  See notes below.
                try:
                    return func(node, capture, filename)
                except Exception as e:
                    exception_queue.put(e)
    
            return inner
    
        def default_query_func(files):
            if selector_func:
                q = selector_func(files)
            else:
                q = Query(files).select(selector)
    
            if modifier_func:
                q = modifier_func(q)
            else:
                q = q.modify(modifier)
    
            return q
    
        if query_func is None:
            query_func = default_query_func
    
        with tempfile.NamedTemporaryFile(suffix=".py") as f:
            # TODO: I'm almost certain this will not work on Windows, since
            # NamedTemporaryFile has it already open for writing.  Consider
            # using mktemp directly?
            with open(f.name, "w") as fw:
                fw.write(input_text + "\n")
    
            query = query_func([f.name])
            assert query is not None, "Remember to return the Query"
            assert query.retcode is None, "Return before calling .execute"
            assert len(query.transforms) == 1, "TODO: Support multiple"
    
            for i in range(len(query.current.callbacks)):
                query.current.callbacks[i] = store_exceptions_on(
                    query.current.callbacks[i]
                )
    
            # We require the in_process parameter in order to record coverage properly,
            # but it also helps in bubbling exceptions and letting tests read state set
            # by modifiers.
            query.execute(
                interactive=False, write=True, silent=False, in_process=in_process
            )
    
            # In the case of in_process=False (mirroring normal use of the tool) we use
            # the queue to ship back exceptions from local_process, which can actually
            # fail the test.  Normally exceptions in modifiers are not printed
            # at all unless you pass --debug, and even then you don't get the
            # traceback.
            # See https://github.com/facebookincubator/Bowler/issues/63
            if not exception_queue.empty():
>               raise AssertionError from exception_queue.get()
E               AssertionError

Thanks for releasing this!

I've been trying to execute multiple steps, and can't seem to find a way.

I could of course create multiple Query() chains but then the diff and file-writing would happen more than once.

Is there a way to chain multiple select&filter clauses in a single query and then execute them?

From reading the bowler source code it seemed like maybe I could do this:

(
    Query()
    .select(...)
    .modify(...)
    .select(...)
    .modify(...)
    .execute(...)
)

but this actually seems to silently overwrite the first select/modify. The second one gets executed correctly but the first disappears into the void.

After installation of pybowler using pip install on windows 10, running bowler in powershell returns TB.
Steps:

1. in Python folder (typically something like C:\Python36_32), go to Scripts folder, and run ".\pip.exe install bowler.
2. run .\bowler.exe in same folder (Python36_32/Scripts)
3. TB as follows:

Traceback (most recent call last):

File "C:\Python36\Scripts\bowler-script.py", line 11, in <module>
  load_entry_point('bowler==0.5.1', 'console_scripts', 'bowler')()

File "c:\python36\lib\site-packages\pkg_resources\__init__.py", line 565, in load_entry_point
  return get_distribution(dist).load_entry_point(group, name)

File "c:\python36\lib\site-packages\pkg_resources\__init__.py", line 2631, in load_entry_point
  return ep.load()

File "c:\python36\lib\site-packages\pkg_resources\__init__.py", line 2291, in load
  return self.resolve()

File "c:\python36\lib\site-packages\pkg_resources\__init__.py", line 2297, in resolve
  module = __import__(self.module_name, fromlist=['__name__'], level=0)

File "c:\python36\lib\site-packages\bowler\__init__.py", line 12, in <module>
  from .tool import BowlerTool

File "c:\python36\lib\site-packages\bowler\tool.py", line 12, in <module>
  import sh

File "c:\python36\lib\site-packages\sh.py", line 36, in <module>
  support." % __version__)

ImportError: sh 1.12.14 is currently only supported on linux and osx. please install pbs 0.110 (http://pypi.python.org/pypi/pbs) for windows support.

There are quite a few, and they deserve a place in the API reference.

Currently, when running bowler do, if IPython isn't available to import, Bowler will just silently exit without saying why.

Culprit code: https://github.com/facebookincubator/Bowler/blob/master/bowler/main.py#L61

I am not sure if I am doing this incorrectly, but here is the use case:

test.py

def f1(z):                                                                                                                                                                                                          
    print('test')                                                                                                                                                                                                   
                                                                                                                                                                                                                    
def f(t, x):                                                                                                                                                                                                        
    f1(z=x)                                                                                                                                                                                                         
                                                                                                                                                                                                                    
f(t=10, x=20)                                                                                                                                                                                                       
               

And then, I use bowler to remove argument t.

q=Query('test.py')

q.select_function('f').remove_argument('t').diff()

But, I get the following error:

   816         stop_at = -1
    817         if "source" not in transform.kwargs:
--> 818             raise ValueError("remove_argument requires passing original function")
    819         signature = inspect.signature(transform.kwargs["source"])
    820         if name not in signature.parameters:

ValueError: remove_argument requires passing original function

Am I missing any steps here?

Parsing python2 code without print functions turned on seems to be broken? The README seems to imply that this should still work (Bowler supports modifications to code from any version of Python 2 or 3)

Dockerfile:

FROM python:3.7

RUN pip3 install git+https://github.com/facebookincubator/Bowler.git
RUN echo "print 'something'" > test.py
RUN bowler dump test.py

Output:

$ bowler dump test.py
ERROR:bowler.tool:Skipping test.py: failed to parse

1. Running a large refactor can cause patch to produce reject files.
2. Bowler doesn't throw an exception when this is the case, which makes it difficult to notice.

Apologies for the size of this issue; I've reduced it as much as I can while still triggering the problem.

This file: https://gist.github.com/craigds/4bb86df3b3ead343c22be6c7c9c14a57
I've snipped it at 1750 lines, because that's just enough to reproduce the issue.

With this script:

from fissix.fixer_util import Newline
from fissix.pygram import python_symbols as syms

from bowler import Query, TOKEN
from bowler.types import Leaf, Node


def Assert(test, message=None, **kwargs):
    """
    Build an assertion statement
    """
    if not isinstance(test, list):
        test = [test]
    test[0].prefix = " "

    return Node(syms.assert_stmt, [Leaf(TOKEN.NAME, "assert")] + test, **kwargs)


def gdaltest_fail_reason_to_assert(node, capture, filename):
    """
    Converts an entire if statement into an assertion.

    if x == y:
        gdal.post_reason('foo')
        return 'fail'

    -->
        assert x != y, 'foo'
    """
    condition = capture["condition"]
    reason = capture["reason"]
    reason = reason.clone()

    assertion = Assert([condition.clone()], reason, prefix=node.prefix)

    # Trailing whitespace and any comments after the if statement are captured
    # in the prefix for the dedent node. Copy it to the following node.
    dedent = capture["dedent"]
    next_node = node.next_sibling
    node.replace([assertion, Newline()])
    next_node.prefix = dedent.prefix


(
    Query("half_of_ogr_fgdb.py")
    .select(
        """
        if_stmt<
            "if" condition=any ":"
            suite<
                any any
                simple_stmt<
                    power<
                        "gdaltest" trailer< "." "post_reason" >
                        trailer< "(" reason=any ")" >
                    >
                    any
                >
                simple_stmt<
                    return_stmt< "return" returntype=STRING >
                    any
                >
                dedent=any
            >
        >
    """
    )
    .modify(callback=gdaltest_fail_reason_to_assert)
    .execute(interactive=False, write=True)
)

output

failed to apply patch hunk
Traceback (most recent call last):
  File "/Users/cdestigter/checkout/decrapify/lib/python3.7/site-packages/bowler/tool.py", line 287, in process_hunks
    sh.patch("-u", filename, _in=patch)  # type: ignore
  File "/Users/cdestigter/checkout/decrapify/lib/python3.7/site-packages/sh.py", line 1427, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/Users/cdestigter/checkout/decrapify/lib/python3.7/site-packages/sh.py", line 774, in __init__
    self.wait()
  File "/Users/cdestigter/checkout/decrapify/lib/python3.7/site-packages/sh.py", line 792, in wait
    self.handle_command_exit_code(exit_code)
  File "/Users/cdestigter/checkout/decrapify/lib/python3.7/site-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1:

  RAN: /usr/bin/patch -u half_of_ogr_fgdb.py

  STDOUT:
patching file half_of_ogr_fgdb.py
Hunk #1 FAILED at 1539.
1 out of 1 hunk FAILED -- saving rejects to file half_of_ogr_fgdb.py.rej


  STDERR:

causes?

Chopping much more out of the source file makes the error not occur. It doesn't seem to matter which parts you chop out.

My hunch is that if the diff gets too large, the patch can no longer match line numbers from old to new, so it rejects the patch.

A workaround might be to pause running fixers when the diff gets bigger than a certain threshold and write the new file out, and then continue running fixers? ie patch in multiple stages.

When transforms or modifiers fail, Bowler prints an error, and stringifies the exception, but this doesn't always help understand what the actual problem is (sometimes it just prints the Node, which is useless). Maybe print a stack trace, or at the very list, print the type of the exception along with the stringified output.

We already have CI for builds and PR's. We should also make it build/deploy pybowler.io on every push to the master branch.

https://docusaurus.io/docs/en/publishing#using-travis-ci

This looks like a great library, but the documentation is not very helpful in teaching it.

I think I need to do a "from bowler import *" to get the examples to work as scripts from the command line, but I don't see that in any docs.

I don't see how to run a query on an module. "bowler run" says it takes a path to a python script, or a module name, but if I do "bowler run my_query.py path.to.module" it still only checks stuff directly under my current directory.

I guess I'm expected to run it by walking the entire tree and doing "bowler run" repeatedly?

This is really useful for validating selectors and modifiers, and would be nice to document for use by others.

Repro

git clone https://github.com/jreese/aql
cd aql
git checkout 3565e740
bowler do  'Query().select_class("Operation").rename("Comparison").diff()'

Expected result

Bowler should generate the diff and exit with success

Actual result

Bowler generates the diff, and then outputs Error: query failed: https://gist.github.com/jreese/0da753bfd9a218d494bd10a7b401ce4f

Enabling debug output does not indicate any error occurred.

Bowler currently defaults to only touching files with a .py extension when given a directory to modify. Bowler should also allow specifying a list of whitelisted extensions, or perhaps some sort of filter function instead, in cases where the files to modify use custom extensions.

I'm just pointing this out:

If I were you, I wouldn't use lib2to3 directly. It has a lot of issues that I fixed in parso. It also has a nicer API, a refactored parser generator that you can actually understand, error recovery and a few more features that might come in handy.

Originally, filename_matcher was intended to just match extensions, but there are reasons you might want to load the file and look at its contents too (#! line, or look for __future__ imports to completely ignore without doing the CST parse). I think this is as simple as passing Filename(fullname) which we have on the following line in refactor_dir already.