Comments (3)
Yep, this is the correct solution.
Parser combinators compose in a way that makes each combinator work without regard to what comes before or after it. In the case of map_with
, the span you get relates to the exact span of input that was parsed by the parser it's called on (not the output of the parser), so removing the padding from the span is as simple as parsing the padding outside the map_with
combinator.
from chumsky.
Then I guess it's not an issue after all. I'm glad my tests include the position information from the spans otherwise I never would have caught this.
from chumsky.
I found a solution but for my issue to keep the spans exactly where they should be. The problem was that I had called padded
throughout my lexer
function and again at the end after mapping the tokens to a span. The correct way is to map all of the tokens to a span before calling padded
so that it does not add extra spaces to your span, which results in slightly wonky-looking error reports. However, it still seems off to me that the span includes extra space when the parser itself is ignoring those characters. Am I missing a rationale for that?
In case someone else has this issue, this is what my lexer
looks like now. Note that there is only one padded
: between each token and that the map_with
is done before the padding.
pub fn lexer<'src>() -> impl Parser<
'src,
&'src str,
Vec<(Token<'src>, SimpleSpan<usize>)>,
extra::Err<Rich<'src, char, SimpleSpan<usize>>>,
> {
let boolean = choice((
just("true").to(Token::Boolean(true)),
just("false").to(Token::Boolean(false)),
));
let float_numeric = just('-')
.or_not()
.then(text::int(10))
.then(just('.').then(text::digits(10)))
.then(just('e').then(text::digits(10)).or_not())
.to_slice()
.map(|text: &str| Token::Float(text.parse().unwrap()));
let float_other = choice((just("Infinity"), just("-Infinity"), just("NaN")))
.map(|text| Token::Float(text.parse().unwrap()));
let float = choice((float_numeric, float_other));
let integer = just('-')
.or_not()
.then(text::int(10))
.to_slice()
.map(|text: &str| {
let integer = text.parse::<i64>().unwrap();
Token::Integer(integer)
});
let delimited_string = |delimiter| {
just(delimiter)
.then(none_of(delimiter).repeated())
.then(just(delimiter))
.to_slice()
.map(|text: &str| Token::String(&text[1..text.len() - 1]))
};
let string = choice((
delimited_string('\''),
delimited_string('"'),
delimited_string('`'),
));
let identifier = text::ident().map(|text: &str| Token::Identifier(text));
let operator = choice((
// logic
just("&&").to(Operator::And),
just("==").to(Operator::Equal),
just("!=").to(Operator::NotEqual),
just(">=").to(Operator::GreaterOrEqual),
just("<=").to(Operator::LessOrEqual),
just(">").to(Operator::Greater),
just("<").to(Operator::Less),
just("!").to(Operator::Not),
just("!=").to(Operator::NotEqual),
just("||").to(Operator::Or),
// assignment
just("=").to(Operator::Assign),
just("+=").to(Operator::AddAssign),
just("-=").to(Operator::SubAssign),
// math
just("+").to(Operator::Add),
just("-").to(Operator::Subtract),
just("*").to(Operator::Multiply),
just("/").to(Operator::Divide),
just("%").to(Operator::Modulo),
))
.map(Token::Operator);
let control = choice((
just("->").to(Control::Arrow),
just("{").to(Control::CurlyOpen),
just("}").to(Control::CurlyClose),
just("[").to(Control::SquareOpen),
just("]").to(Control::SquareClose),
just("(").to(Control::ParenOpen),
just(")").to(Control::ParenClose),
just(",").to(Control::Comma),
just(";").to(Control::Semicolon),
just("::").to(Control::DoubleColon),
just(":").to(Control::Colon),
just("..").to(Control::DoubleDot),
just(".").to(Control::Dot),
))
.map(Token::Control);
let keyword = choice((
just("any").to(Keyword::Any),
just("bool").to(Keyword::Bool),
just("break").to(Keyword::Break),
just("else").to(Keyword::Else),
just("float").to(Keyword::Float),
just("int").to(Keyword::Int),
just("if").to(Keyword::If),
just("list").to(Keyword::List),
just("map").to(Keyword::Map),
just("none").to(Keyword::None),
just("range").to(Keyword::Range),
just("struct").to(Keyword::Struct),
just("str").to(Keyword::Str),
just("loop").to(Keyword::Loop),
just("while").to(Keyword::While),
))
.map(Token::Keyword);
choice((
boolean, float, integer, string, keyword, identifier, control, operator,
))
.map_with(|token, state| (token, state.span()))
.padded()
.repeated()
.collect()
}
from chumsky.
Related Issues (20)
- `ContainerExactly` impls for `Rc` and `Arc` are unsound HOT 2
- Clone trait bounds not satisfied HOT 2
- Trait bound not satisfied on foldl HOT 2
- Input lifetimes HOT 4
- bug: couldn't build `examples/foo.rs` HOT 1
- Query : Create a parser that parses a vec of patterns HOT 7
- Crashed while parsing markup text HOT 4
- Fix SpannedInput HOT 1
- Trying to run the `nano_rust.rs` example using `chumsky = "1.0.0-alpha.6"` HOT 4
- Question: Using `map_with` to get spans HOT 2
- Question: Writing a Ternary Operator Parser HOT 5
- Can't build example HOT 1
- `/examples/pythonic.rs` is commented out HOT 1
- `/examples/foo.rs` is different from the tutorial HOT 1
- stack overflow HOT 3
- Ability to create warnings HOT 2
- `vars` gets not correctly truncated in example code HOT 1
- error: "unknown feature stdsimd" cause by the dependency of ahash when update rustc to 1.78.0-nightly HOT 2
- Cannot use custom span type with Stream HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chumsky.