`separated_by` does not work as expected about chumsky HOT 6 CLOSED

@zesterer I didn't realized it was on master already! I was reading the doc on https://docs.rs, and that's why I didn't find it. Thanks a ton!

from chumsky.

Oh, just realized that it has been fixed in #25. I should have checked the master branch before posting it here. I'm closing this issue now.

from chumsky.

Sorry I have to re-open this issue. Although the problem I described in the OP has all worked as expected in @ryo33's fix, I found that separate_by still has a problem when combined with alternation.

Let's look at the code:

use chumsky::prelude::*;

fn main() {
  let a_list = just::<char, Simple<_>>('a').separated_by(just(','));
  let b_list = just('b').separated_by(just(','));

  let parser = a_list.or(b_list);

  // works as expected
  assert_eq!(parser.parse("a"), Ok(vec!['a']));

  // expected: ['b']
  // actual: []
  assert_eq!(parser.parse("b"), Ok(vec!['b']));
}

from chumsky.

Let's look at the code

This is correct behaviour. Parsers are lazy: they'll only parse the input you tell them to parse and no more. Some combinators (.repeated(), .separated_by(...), .or_not(), etc.) are able to parse no input at all and still produce a valid result.

For example, consider a parser that parses an array:

let array = expr
    .separated_by(just(','))
    .delimited_by('[', ']');

When given the input [] this parse succeeds! This only happens because the inner .separated_by(...) parser consumed no input (because there were no array items).

Obviously, when you're parsing a full program, you generally don't want the parser to consume only the first valid pattern and ignore the rest. To solve this, you can 'force' the parser to parse input right the way to the end using .then_ignore(end()).

For example, just('a').repeated() will succeed on the input "bbb" because it parses no characters and leaves the rest. However, just('a').repeated().then_ignore(end()) will not because it's expecting the end of the input afterwards.

from chumsky.

Thanks, your explanation makes total sense to me.

It seems to me what I need is an option that sort of act like Repeated::at_least. For now I'm going to use try_map at the end to enforce the number of repetition. But ideally I hope SeparatedBy can behave similarly to Repeated that has these options (at_least, at_most) to be play with.

I'll see if I can send a PR for that when I finished what I got on my hand :)

from chumsky.

You can do foo.separated_by(...).at_least(n). Is this what you mean?

from chumsky.