Ensure top community D projects parse successfully about tree-sitter-d HOT 11 OPEN

IASM GCC Grammar:

GccAsmStatement:
    asm FunctionAttributes(opt) { GccAsmInstructionList }

GccAsmInstructionList:
    GccAsmInstruction ;
    GccAsmInstruction ;  GccAsmInstructionList

GccAsmInstruction:
    GccBasicAsmInstruction
    GccExtAsmInstruction
    GccGotoAsmInstruction

GccBasicAsmInstruction:
    AssignExpression

GccExtAsmInstruction:
    AssignExpression : GccAsmOperands(opt)
    AssignExpression : GccAsmOperands(opt) : GccAsmOperands(opt)
    AssignExpression : GccAsmOperands(opt) : GccAsmOperands(opt) : GccAsmClobbers(opt)

GccGotoAsmInstruction:
    AssignExpression :  : GccAsmOperands(opt) : GccAsmClobbers(opt) : GccAsmGotoLabels(opt)

GccAsmOperands:
    GccSymbolicName(opt) StringLiteral ( AssignExpression )
    GccSymbolicName(opt) StringLiteral ( AssignExpression ), GccAsmOperands

GccSymbolicName:
    [ Identifier ]

GccAsmClobbers:
    StringLiteral
    StringLiteral , GccAsmClobbers

GccAsmGotoLabels:
    Identifier
    Identifier , GccAsmGotoLabels

from tree-sitter-d.

Any thoughts?

Probably yes on all three counts.

from tree-sitter-d.

How is GCC-style assembly actually used in the wild? If I understand correctly, you can't just put it inside a version(GDC) block since code in there still has to parse, so I suppose people either:

Nope, you put it in a version condition. The parser just collects all the tokens within braces, and then the semantic parses the contents.

from tree-sitter-d.

@ibuclaw https://github.com/libmir/mir-cpuid/blob/4add5b639351f15044a8c991ce3a905d7973c3de/source/cpuid/x86_any.d#L606-L609 still doesn't parse with the grammar you posted, is that expected?

That's deprecated syntax, and will be an error soon™

from tree-sitter-d.

@ibuclaw Thanks!

So, that raises the question of where this should be kept / maintained:

• Obviously we could keep it as a local patch in this project, but that won't make it very useful for other applications, and it would need to be updated here in case of changes.
• Perhaps it could just be added to dlang.org, as an informative section. Grammar consumers could choose if they want to use this block by adding GccAsmStatement as a possible option for AsmStatement, OSLT.
• Alternatively, the GDC website could have a page with GDC-specific language changes/extensions. We could use the same machinery used by dlang.org to render it.

Any thoughts?

from tree-sitter-d.

How is GCC-style assembly actually used in the wild? If I understand correctly, you can't just put it inside a version(GDC) block since code in there still has to parse, so I suppose people either:

• only support compiling with GDC
• make separate files for GCC-style assembly and DMD assembly and let the build system pass the right one to the compiler
• put the GCC-style assembly code is inside a string mixin versioned on GCC

Currently iasm.d looks like:

version (MARS)
{
    import dmd.iasmdmd;
}
else version (IN_GCC)
{
    import dmd.iasmgcc;
}

Can't both compilers at least support parsing both, and have GccAsmStatement in their grammar? Or is there ambiguity?

from tree-sitter-d.

@ibuclaw https://github.com/libmir/mir-cpuid/blob/4add5b639351f15044a8c991ce3a905d7973c3de/source/cpuid/x86_any.d#L606-L609 still doesn't parse with the grammar you posted, is that expected?

from tree-sitter-d.

Can't both compilers at least support parsing both, and have GccAsmStatement in their grammar? Or is there ambiguity?

Looks like LDC does a lazy check on the first token for the easy cases where it cannot be a dmd-style iasm statement. This would mean that there are some styles of asm that LDC does not support.

asm { ctfeFunctionReturningInstructionsAsString(); }

There are a number of ambiguities between an dmd-style AsmExp and the AssignExpression instruction string of the gdc-style, i.e:

enum cpuid = "cpuid";
asm { cpuid; } // accepted dmd-style asm

These could perhaps be overcome by tweaking the grammar to force instruction expressions (not string literals) to be enclosed in parentheses in the gdc style.

from tree-sitter-d.

A small change of plans: ccfcf13

from tree-sitter-d.

I would encourage looking at https://github.com/gdamore/tree-sitter-d

I believe it parses pretty much everything correctly.

I've tested it with the DMD test suite, with DMD itself, with DUB, and a large body of non-public work.

from tree-sitter-d.

Hi @gdamore,

First, thanks for taking the initiative in putting together a complete tree-sitter package for D!

That said, please forgive me if I'm a little skeptical about your claims. Looking at the list in this issue's description, many instances of errors produced by the implementation in this repository is not due to defects in the implementation, but genuinely because the source files contain invalid grammar. If you're claiming that they parse successfully, that would indicate that your parser is overly relaxed, and will consume invalid grammatical constructs and present them as valid D programs.

I'm also puzzled by the route you've taken for your project. As I understand, your tree-sitter grammar is hand-written. Was there any reason why you didn't just take the output from this project and added the missing pieces, such as highlighting queries, to produce a complete package usable in editors?

The main distinction between this project and your endeavor is the scope. This project aims to:

• produce a tree-sitter grammar that's usable in text editors for syntax highlighting (as is your project)
• ensure the official D grammar is well-formed, and generally machine-readable
• find discrepancies in the official D grammar and fix them appropriately (see #4)
• find discrepancies / corner cases in the compiler frontend (DMD) and fix them appropriately (this hasn't been done yet in scale but we might be able to do it by generating random programs with the grammar)
• continuously test the compiler test suite to ensure it conforms to the official grammar (which guarantees that all language changes are properly documented in the language specification)
• ensure the tree-sitter grammar is "ever-green" and constantly stays in sync with the language specification and implementation.

The last point is an important one. Tools will go out of date as soon as someone stops actively looking at them. For evidence of how much this affects our community, here is an example: https://forum.dlang.org/post/[email protected]

As such, I really think your efforts would have much more impact by contributing to the effort started here. Unfortunately due to recent circumstances I've been unable to dedicate as much time to my open source projects as I would have liked, but I remain available for questions and support.

from tree-sitter-d.