Feature request: Ignore comments, don't strip them out about sqlfmt HOT 5 CLOSED

I would love to support comments instead of remove them. They are hard because our parser always removes them. But I think we have a few workarounds.

For your use case, which is comments in between statements, we can maybe do something to support those using the lexer to look for comments on either side of a SQL semicolon token and print them somehow. I feel like this is doable because I think it can be done without changing the current lexer or parser (or maybe a simple change to the lexer to have it emit comments? Unsure).

The more difficult version is to support comments in the middle of a statement. This would require teaching AST nodes about comments or else some shadow data structure that could attach comments to any node in the AST somehow, which would have to be consulted during pretty printing. I think this feature would be much harder, because I'm unsure how to do it.

Thoughts @knz?

from sqlfmt.

Going the cockroachdb lexer route is going to be hard and inelegant.

Mainly because the lexer code is complex, and adding special code on a lexer that is otherwise optimized for speed in the production case, just for satisfying the needs of sqlfmt, feels like the wrong choice.

I think the only sane choice to achieve this is to pre-process using a separate tool / lexer.

from sqlfmt.

For example I would simply split the input at newlines, detect lines containing only comments (top level), and then feed the lines in-between comments separately to sqlfmt.

from sqlfmt.

I don't think that works for queries like

select '
-- 1
2';

from sqlfmt.

that is not a query that interests rich. I do not care about this type of things. I would be relatively opposed to making the crdb scanner more complex just so that sqlfmt has correct behavior in those odd cases.

If that is important to you I fear the only reasonable option is to fork the scanner into the sqlfmt tool (and ensure it doesn't get linked with the crdb binary so we don't have code duplication).

from sqlfmt.