Black cannot parse previously parseable file in 24.4.1 about black HOT 14 CLOSED

No no, thank YOU! We love your tool :) (Well, I love it. My team grumbles when they forget to run it and our linting job snarks at them.)

from black.

I got something that appears to work: #4332.

from black.

Thanks! cc @tusharsadhwani.

from black.

Using the same quotes is a syntax error in Python itself. You terminate the string early.

>>> f'{1:{f'{2}'}}'
  File "<stdin>", line 1
    f'{1:{f'{2}'}}'
            ^
SyntaxError: f-string: expecting '}'

from black.

@tarper24 it works fine on Python 3.12 onwards.

from black.

@tarper24 not in Python 3.12 any more. That's actually why we made this change; we had to revamp the parser around f-strings to support the new syntax. Unfortunately that caused us to start failing on some f-strings that were already valid. We found a few such cases before release by running Black on various codebases, but unfortunately we missed your case.

from black.

This seems like the minimal reproduction:

f"{1:{f'{2}'}}"

from black.

actually, using same or different quotes gives us two different crash scenarios:

f'{1:{f'{2}'}}'

If the quotes of the outer and inner fstring are the same, we get a different crash.

from black.

I spent some time on this but couldn't figure out a solution yet.

The reproducer gets tokenized like this:

% python -m blib2to3.pgen2.tokenize 4329.py
1,0-1,2:	FSTRING_START	'f"'
1,2-1,2:	FSTRING_MIDDLE	''
1,2-1,3:	LBRACE	'{'
1,3-1,4:	NUMBER	'1'
1,4-1,5:	OP	':'
1,5-1,5:	FSTRING_MIDDLE	''
1,5-1,6:	OP	'{'
1,6-1,8:	FSTRING_START	"f'"
1,8-1,8:	FSTRING_MIDDLE	''
1,8-1,9:	LBRACE	'{'
1,9-1,10:	NUMBER	'2'
1,10-1,11:	OP	'}'
1,11-1,12:	FSTRING_MIDDLE	"'"
1,12-1,13:	RBRACE	'}'
Traceback (most recent call last):

The FSTRING_MIDDLE "'" near the end is wrong; it should be an FSTRING_END, closing the inner f-string.

My current thinking is that the issue is that the inside_fstring_colon in the tokenizer gets set to True for the outer f-string and then applied incorrectly while we're parsing the inner f-string. To address that, I tried turning inside_fstring_colon into a stack with an entry for each nested f-string, but that so far doesn't work.

from black.

Commenting out and bracelev == 0 in the part that yields RBRACE fixes this case. But it breaks other cases. That's how far I got yesterday night

from black.

Also it's not the FSTRING_MIDDLE that's incorrect, it's the OP just above it, which should be an RBRACE to match the LBRACE.

from black.

The minimised case that breaks when making the bracelev change is:

f'{1:{2}d}'

from black.

What is the difference between OP and LBRACE/RBRACE here? I noticed the variation but it wasn't clear to me which one is correct.

from black.

In the original impl it's very blurry what to use, but I went with yielding LBRACE whenever we go from collecting FSTRING_MIDDLE tokens to parsing python expressions again

from black.