Code Monkey home page Code Monkey logo

Comments (3)

zesterer avatar zesterer commented on June 12, 2024 1

Yep, this is the correct solution.

Parser combinators compose in a way that makes each combinator work without regard to what comes before or after it. In the case of map_with, the span you get relates to the exact span of input that was parsed by the parser it's called on (not the output of the parser), so removing the padding from the span is as simple as parsing the padding outside the map_with combinator.

from chumsky.

solaeus avatar solaeus commented on June 12, 2024 1

Then I guess it's not an issue after all. I'm glad my tests include the position information from the spans otherwise I never would have caught this.

from chumsky.

solaeus avatar solaeus commented on June 12, 2024

I found a solution but for my issue to keep the spans exactly where they should be. The problem was that I had called padded throughout my lexer function and again at the end after mapping the tokens to a span. The correct way is to map all of the tokens to a span before calling padded so that it does not add extra spaces to your span, which results in slightly wonky-looking error reports. However, it still seems off to me that the span includes extra space when the parser itself is ignoring those characters. Am I missing a rationale for that?

In case someone else has this issue, this is what my lexer looks like now. Note that there is only one padded: between each token and that the map_with is done before the padding.

pub fn lexer<'src>() -> impl Parser<
    'src,
    &'src str,
    Vec<(Token<'src>, SimpleSpan<usize>)>,
    extra::Err<Rich<'src, char, SimpleSpan<usize>>>,
> {
    let boolean = choice((
        just("true").to(Token::Boolean(true)),
        just("false").to(Token::Boolean(false)),
    ));

    let float_numeric = just('-')
        .or_not()
        .then(text::int(10))
        .then(just('.').then(text::digits(10)))
        .then(just('e').then(text::digits(10)).or_not())
        .to_slice()
        .map(|text: &str| Token::Float(text.parse().unwrap()));

    let float_other = choice((just("Infinity"), just("-Infinity"), just("NaN")))
        .map(|text| Token::Float(text.parse().unwrap()));

    let float = choice((float_numeric, float_other));

    let integer = just('-')
        .or_not()
        .then(text::int(10))
        .to_slice()
        .map(|text: &str| {
            let integer = text.parse::<i64>().unwrap();

            Token::Integer(integer)
        });

    let delimited_string = |delimiter| {
        just(delimiter)
            .then(none_of(delimiter).repeated())
            .then(just(delimiter))
            .to_slice()
            .map(|text: &str| Token::String(&text[1..text.len() - 1]))
    };

    let string = choice((
        delimited_string('\''),
        delimited_string('"'),
        delimited_string('`'),
    ));

    let identifier = text::ident().map(|text: &str| Token::Identifier(text));

    let operator = choice((
        // logic
        just("&&").to(Operator::And),
        just("==").to(Operator::Equal),
        just("!=").to(Operator::NotEqual),
        just(">=").to(Operator::GreaterOrEqual),
        just("<=").to(Operator::LessOrEqual),
        just(">").to(Operator::Greater),
        just("<").to(Operator::Less),
        just("!").to(Operator::Not),
        just("!=").to(Operator::NotEqual),
        just("||").to(Operator::Or),
        // assignment
        just("=").to(Operator::Assign),
        just("+=").to(Operator::AddAssign),
        just("-=").to(Operator::SubAssign),
        // math
        just("+").to(Operator::Add),
        just("-").to(Operator::Subtract),
        just("*").to(Operator::Multiply),
        just("/").to(Operator::Divide),
        just("%").to(Operator::Modulo),
    ))
    .map(Token::Operator);

    let control = choice((
        just("->").to(Control::Arrow),
        just("{").to(Control::CurlyOpen),
        just("}").to(Control::CurlyClose),
        just("[").to(Control::SquareOpen),
        just("]").to(Control::SquareClose),
        just("(").to(Control::ParenOpen),
        just(")").to(Control::ParenClose),
        just(",").to(Control::Comma),
        just(";").to(Control::Semicolon),
        just("::").to(Control::DoubleColon),
        just(":").to(Control::Colon),
        just("..").to(Control::DoubleDot),
        just(".").to(Control::Dot),
    ))
    .map(Token::Control);

    let keyword = choice((
        just("any").to(Keyword::Any),
        just("bool").to(Keyword::Bool),
        just("break").to(Keyword::Break),
        just("else").to(Keyword::Else),
        just("float").to(Keyword::Float),
        just("int").to(Keyword::Int),
        just("if").to(Keyword::If),
        just("list").to(Keyword::List),
        just("map").to(Keyword::Map),
        just("none").to(Keyword::None),
        just("range").to(Keyword::Range),
        just("struct").to(Keyword::Struct),
        just("str").to(Keyword::Str),
        just("loop").to(Keyword::Loop),
        just("while").to(Keyword::While),
    ))
    .map(Token::Keyword);

    choice((
        boolean, float, integer, string, keyword, identifier, control, operator,
    ))
    .map_with(|token, state| (token, state.span()))
    .padded()
    .repeated()
    .collect()
}

from chumsky.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.