(Helpful) Error messages about pe HOT 3 OPEN

keep the original grammar as well and if there is a parse error re-parse with the unoptimized grammar in order to provide more helpful error messages.

Pe already stores the original and optimized grammars in the parser objects:

The intended reason for storing both is precisely what you describe, however I have not yet implemented the reparsing feature. One challenge is that I don't see an easy way to hand over parser state at a point to the unoptimized grammar. I can reparse the whole input, but that can take a while, especially with the unoptimized grammar.

from pe.

I agree, and I think better error messages are the thing most needed by this project.

If you use the packrat parser, you will get close to what you are asking with the pe.STRICT flag:

>>> pe.match('"a"+ "b"+', "aacc", flags=pe.STRICT)
Traceback (most recent call last):
  [...]
pe._errors.ParseError: ParseError: failed to parse; use memoization for more details

Also use the pe.MEMOIZE flag to get more precise error messages:

>>> pe.match('"a"+ "b"+', "aacc", flags=pe.STRICT|pe.MEMOIZE)
Traceback (most recent call last):
  [...]
pe._errors.ParseError: 
  line 0, character 0
    aacc
    ^
ParseError: `(?=(?P<_1>a+))(?P=_1)(?=(?P<_2>b+))(?P=_2)`

The next issue is that the patterns are optimized, which means:

1. Multiple patterns/rules get merged into a single terminal regex, making pe unable to see parse failures internal to the regex (thus the ^ points to the first a and not the first c)
2. The optimized regex is hard to follow and doesn't resemble the original grammar

To overcome this, you will need to first compile the grammar and then match, because the flags option of pe.match() is only for the matching and not the compiling (the compilation and matching flags are currently conflated... I may separate them in a future version):

>>> p = pe.compile('"a"+ "b"+', parser="packrat", flags=pe.NONE)
>>> p.match("aacc", flags=pe.STRICT|pe.MEMOIZE)
Traceback (most recent call last):
  [...]
pe._errors.ParseError: 
  line 0, character 2
    aacc
      ^
ParseError: `a`, `b`

Another strategy is to use the pe.DEBUG flag at compilation time, which is also currently only available for the packrat parser:

>>> p = pe.compile('"a"+ "b"+', parser="packrat", flags=pe.DEBUG)
## Grammar ##
Start <- "a"+ "b"+
>>> p.match("aacc", flags=pe.STRICT|pe.MEMOIZE)
aacc         |    "a"+ "b"+
aacc         |      "a"+
aacc         |        "a"
acc          |        "a"
cc           |        "a"
cc           |      "b"+
cc           |        "b"
Traceback (most recent call last):
  [...]
pe._errors.ParseError: 
  line 0, character 2
    aacc
      ^
ParseError: `a`, `b`

In a terminal that supports ANSI colors, you'll see the terminals that failed in red and those that succeeded in green.

Getting good error messages from a general-purpose recursive descent parser is a significant challenge in itself, but doing that when you have an optimized grammar is even more difficult. I welcome any help in this area.

Since the machine parser does not yet do memoization or debug rules, a strategy for grammar development is to test things out with the packrat parser, then switch to the machine parser for speed once it's working.

from pe.

Yeah, producing good error messages is incredibly difficult.

For the issue with having an optimized grammar, one potential workaround could be to keep the original grammar as well and if there is a parse error re-parse with the unoptimized grammar in order to provide more helpful error messages. That should probably be optional though due to the additional overhead of parsing the same input twice.

from pe.