Non discarding API about bread-n-butter HOT 3 CLOSED

I'm not sure I understand why you'd use bnb to implement a lexer, it's meant to be an all-in-one parser like Parsimmon without a separate lexing step.

I thin the amount of complication this would add to the API is not worth it considering you can manually track the "whitespace" of your language and use other combinators like .many() or whatever.

from bread-n-butter.

I'm sorry if lexing is not a potential use case of bnb (and Parsimmon). It's common for complex languages to have separate lexing and (semantical) parsing steps; it's also not unheard to only lex the input without actually parsing it (syntax highlighters and linters for example).

I thin the amount of complication this would add to the API is not worth it considering you can manually track the "whitespace" of your language and use other combinators like .many() or whatever.

I think the misconception here is, the separators discarded by sepBy can be potentially non-trivial. For instance, an arithmetic expression (e.g. 3+4-5*6) can be expressed by operand.sepBy(operator), but the operators are important tokens to keep.

sepBy itself surely can be replaced with other combinators (although cumbersome), I think the question here is where do we draw the line between a core feature (should be included in the lib) and a utility (may not be included).

btw this is actually an extension of the discussion in jneen/parsimmon#230, where I first addressed this issue. I don't mind if bnb doesn't have the non-discarding APIs (we implemented our language without them anyways), but I hope they can appear in Parsimmon v2.

from bread-n-butter.

I'm well aware of lexing as a separate step, it's just not part of bnb intentionally.

If you're looking for more control over various parsers, that's why I allow custom parser creation. I tried to make the custom parser API easier than Parsimmon, and I intentionally tried to keep the source code easier to read rather than speed optimized so folks can see how the library works.

Between custom parsers and .thru, I'm only going to add things to bnb if I want to. It's easy to extend or replace a lot of bnb, and right now working on this library is not my top priority.

from bread-n-butter.