Optional end to chunks

We probably want to hold off on jamming style information into the AST until we have complete grammar coverage.

How can this be done in a way that isn't too cumbersome?

A sample block, with all the places we care about whitespace being marked:

<HERE>if <expression><HERE>then
    <chunk>
<HERE>elseif <expression><HERE> then
    <chunk>
<HERE>else
    <chunk>
<HERE>end

Since we always consume whitespace before a token, any whitespace before <expression> or <chunk> belongs to those AST nodes instead, which simplifies things a bit.

Having to have an EndOfFile token is awkward. It might make more sense if the parser rules are a little smarter and comments and whitespace appeared in stream, next to tokens.

Tokens would have another layer added to them, something like:

enum Token<'a> {
    Semantic(ActualToken<'a>),
    Whitespace(Cow<'a, str>),
    Comment(Comment<'a>),
}

It should be possible to validate that the given AST is valid for a given version of Lua by casting between ASTs.

For example:

let ast = parse(source);
let ast_5_1 = ast.to_version_5_1().expect("Source was not valid Lua 5.1! (maybe it used goto?)");
let new_ast = ast_5_1.to_version_agnostic(); // What do we call this?

assert_eq!(ast, new_ast);

I had some conversations about how to structure parsing of local declarations, and I think there should be one rule that branches to both local variables and local functions.

I'm on mobile from bed so I can't go into details, but I wanted to get this down before I forgot! 😅

Right now test coverage for Rust is kind of a mess, but it's possible to get it working on Linux, which we're building tests on. This thread has a tutorial on collecting test coverage; the first reply has a .travis.yml snippet for integrating kcov, travis, and coveralls.io. It might be nice to get some coverage metrics working early, so we can keep track of what needs to be tested and what's okay.

Once the grammar is finished, we should start tracking how quick the parser is.

I hope it's fast?

Lua has an official testing suite for 5.1 to 5.3:

https://www.lua.org/tests/

It'd be neat if mab tracked its coverage over each version of Lua (#18) or for the time being of the 5.1 test suite.

I'll keep this issue open until it's done, and start filling out missing features once the emitter can do anything at all.

This is a departure from Lua syntax, where a trailing comma in an expression list is a parse error:

print(1, 2,)

ParseExpressionList does not handle this right now; it will parse a trailing comma perfectly fine.

Right now, the test system is a little bit unintuitive. For contributors, it would be useful to document the expected flow when working on the parser.

I don't know what to call this project. Suggestions? Something moon related?

Error reporting coverage is pretty spotty, and the idea probably needs to be revisited.

In rough order of importance:

• Precedence (I've been putting this off -- simple explanation of precedence climbing/Pratt parsing)
• Assignment (depends on varlist)
• Function Declarations
  • Correct function name declaration (currently only identifiers are allowed)
  • Vararg
  • Anonymous function expression
• String literals
  • Regular single and double quote forms
  • Long-form strings
• Comments
  • Infrastructure in tokens for comments
  • Single line comments
  • Multi-line comments
• Generic For (for-in)
• Remaining binary operators
• Function call without parentheses
• Semicolon as statement separator
• ParenExpression ('(' expression ')')
• Lua 5.2 syntax
  • Goto and labels
• Lua 5.3 syntax
  • Bitwise operators