carljm / compgenerator Goto Github PK

We still filter out a lot of generated samples as non-compilable or not containing a listcomp. If we could constrain generation better and filter less (without excluding interesting examples), it would make the fuzzer run a lot faster.

A couple of things we should consider changing:

• Use __a and __b as the names, so we can catch name mangling bugs
• Constrain what kind of expressions appear in subscripts, so we generate less code like (1)[a] that will fail at runtime. I feel such code is less likely to be interesting, as the TypeError from the subscript could hide more interesting errors.
• In the function evaluation code from #2, first check how many arguments the function takes using inspect, and pass those arguments. That way, we get a bit more coverage of locals.

We use recursive strategies in the AST. Currently we have an ugly hack where we first define simple strategies for List and Tuple that can only contain names and constants, and then later define more complex strategies that can include other expressions. This would be better handled using https://hypothesis.readthedocs.io/en/latest/data.html#hypothesis.strategies.recursive

Also, the inconsistency between use of st.from_type(NodeType) vs directly referring to a strategy function by name had something to do with working around recursion.

The fuzzer found this (slightly simplified):

def a(b):
    [(1) for b in b if (a := a)]
a("a")

On main:

>>> a("a")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in a
UnboundLocalError: cannot access local variable 'a' where it is not associated with a value

On 3.11:

>>> a("a")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in a
  File "<stdin>", line 2, in <listcomp>
NameError: cannot access free variable 'a' where it is not associated with a value in enclosing scope

I think that's expected: previously a walrus like this would create an implicit nonlocal, but now it's just a local. Perhaps this should be documented somewhere, though it's hard to imagine real code being affected.

For the fuzzer, we should somehow ignore this difference; I'll try to think of a way.