Preprocessor: Overly aggressive macro expansion? about arocc HOT 3 CLOSED

6.10.3.4 Rescanning and further replacement
After all parameters in the replacement list have been substituted and # and ## processing has
taken place, all placemarker preprocessing tokens are removed. The resulting preprocessing token
sequence is then rescanned, along with all subsequent preprocessing tokens of the source file, for
more macro names to replace.
If the name of the macro being replaced is found during this scan of the replacement list (not
including the rest of the source file’s preprocessing tokens), it is not replaced. Furthermore, if any
nested replacements encounter the name of the macro being replaced, it is not replaced. These
nonreplaced macro name preprocessing tokens are no longer available for further replacement even
if they are later (re)examined in contexts in which that macro name preprocessing token would
otherwise have been replaced.

Pasting the relevant piece of the standard here for future reference. Going from this description to the actual algorithm is very tricky, a lot is implict (in my opinion, it's borderline underspecified).

In thi case, this is how i would imagine the expansion sequence to be like:

^|REC_1$
^REC_1|$
^|REC_DEFER(REC_0_HOOK)()$
^REC_DEFER|(REC_0_HOOK)()$
^|REC_0_HOOK REC_EMPTY()$
^REC_0_HOOK| REC_EMPTY()$
REC_0_HOOK^ REC_EMPTY|()$
REC_0_HOOK^|()$
REC_0_HOOK^()|$
REC_0_HOOK()^|$

legend:

• ^ is start_idx+advance_idx
• $ is moving_end_idx
• | is idx
  where I'm referring to the variables used in the expansion and scanning phase in Preprocessor.zig
  So, initially my code didn't have the advance_idx concept. The sequence was always rescanned beginning from the first expansion point.
  The standard is not very clear about this point, but inferring from gcc and clang, I decided to advance the start_idx when all the tokens to its left are expanded and there's no opportunity of further expanding them in the current context.
  The relevant logic is:
  

  arocc/src/Preprocessor.zig

  Lines 953 to 955 in 76a1066

  	if (idx - start_idx == advance_index + 1 and !do_rescan) {
  	advance_index += 1;
  	}

  
  And this is correct.
  Unfortunately, when the preprocessor finds REC_0_HOOK, it sees that is could be a macro call, but in the end it turns out it isn't because there are no () after it, so this is the codepath taken:
  

  arocc/src/Preprocessor.zig

  Lines 876 to 878 in 76a1066

  	const args = (try pp.collectMacroFuncArguments(tokenizer, buf, &macro_scan_idx, &moving_end_idx, extend_buf, macro.is_builtin)) orelse {
  	idx += 1;
  	continue;

  
  The continue prevents the logic that increases advance_idx to be run.

I'm pretty busy with uni atm, if anybody else wants to pick up the fix from here feel free to do so.

from arocc.

Weird, I had fixed a very similar test case when rewriting the expansion logic.
I will look into it tomorrow, I hope it's not the usual undefined behavior expansion

from arocc.

Here is the ppstep output in case that helps:

pp (started)> s
[preproc.c:8:1]: REC_1
pp (called)> s
[preproc.c:8:1]: REC_DEFER ( REC_0_HOOK ) ( )
pp (expanded)> s
[preproc.c:8:1]: REC_DEFER ( REC_0_HOOK ) ( )
pp (called)> s
[preproc.c:8:1]: REC_0_HOOK REC_EMPTY ( )
pp (expanded)> s
[preproc.c:8:1]: REC_0_HOOK REC_EMPTY ( )
pp (called)> s
[preproc.c:8:1]: REC_0_HOOK   ( )
pp (expanded)> s
[preproc.c:8:1]: REC_0_HOOK ( )
pp (rescanned)> s
[preproc.c:8:1]: REC_0_HOOK ( )
pp (rescanned)> s
[preproc.c:8:6]: REC_0_HOOK ( )

pp (lexed)> s
[preproc.c:10:1]: REC_0_HOOK ( )

pp (complete)> s

from arocc.