elegaanz / rustre Goto Github PK

2 solutions :

1. We ditch strings altogether. That's radical but not necessarily impossible, it mostly depends on rowan's and logos' ability to work with byte slices. That's why Paths exist in Rust in the first place : they may not be UTF-8. Clearly the best solution if it's doable.
2. We convert files before parsing them. That's unfortunately a bit slower and IMO a little more undesirable as we're clearly transforming user input just because we like it better. That also prevents us from pretending to be 100% compatible with lustres's spec. And this also requires asking the user which encoding their files are in, which is a new CLI parameter/function argument

Edit: byte slice support

• logos: should be good in theory, but I don't know if regexes work well with non-string input
• rowan: unfortunately, the GreenNode::new constructor takes an &str. We can try to open an issue, but if this doesn't work, it's always possible to cheat. We could for instance encode the byte slice as base64 and prepend a discriminator char. Dirty, but still better than option 2 IMO as we're not imposing our own rules.

The current algorithm works well but is terribly slow. Another algorithm that should work as good and be much faster :

start = 0
end = 1
while start < src.len()
    matches = tokenize(src[start..end])
    if matches.len() == 1
        push matches[0]
        start = end
        end = start + 1
    else if matches.len() == 0
        error "unexpected character"
    else
        end = end + 1

That associates an identifier with a defintion

Blocked by #4