Comments (4)
Glad to hear that you've found RLTK useful. Do you mind if I ask what kind of project you are using this for?
Currently there is no functionality in the Lexer class for accessing capture groups in a rule match. This is because you would usually emit multiple tokens in the lexer and then have a rule in the parser that would produce the desired AST node.
Continuing your example, you would want to emit a PERIOD, STRING, and LBRACE token in separate rules and then have a production in your parser something like array_access -> PERIOD STRING LBRACE expression RBRACE
. The advantage of this method is that it simplifies your lexer and leaves the heavy lifting to the parser. It also means that you give more information to the parser, which it can then use to disambiguate cases when parsing.
Another solution, which I'd be fine with implementing, is to make the match object available in the Lexer::Environment instance in which the productions are evaluated.
Anyway, I hope that helps.
from rltk.
Hm, okay, that kind of makes sense. I'm prototyping a shell language that relies heavily on whitespace-delimited expressions, so for example
.foo [a b c]
is two expressions, while
.foo[a b c]
is just one.
Currently the lexer has a /\s+/
rule that has no action, so if I used three different tokens, the parser would have no way of differentiating between those two. I suppose I could have a :WHITESPACE
token or some such, but it's a lot simpler just to grab the whole thing and extract the name.
Exposing the match object in Lexer::Environment
seems like a good solution. And if there's any way to make the perly regex pseudoglobals ($1
, $@
, etc) available in the action's scope, that'd be awesome too.
from rltk.
So I've just pushed some changes that give you access to the MatchData object inside the Lexer::Environment object through the match
accessor. I'm a firm believer that global variables are evil so I'll leave exposing the pseudoglobals up to you. Remember that you can extend the Lexer::Environment
class in your lexer fairly simply.
Since whitespace has semantic meaning in your language I would still recommend emitting the :WHITESPACE
token. Lexers are usually used to handle syntax, while semantics are usually handled in the parser. I think if you plan on adding features to this language you'll find it easier to add them in the parser then in the lexer, and you can only do that if you're emitting :WHITESPACE
tokens.
P.S. How did you come across RLTK? I'm trying to promote it better and it would be good to know how people have found it so far.
from rltk.
Good points all. I'll start emitting :WHITESPACE
tokens, then. I found RLTK through a search on rubygems.org for "lexer" - it was a few items down, but it's the only project I found that seemed to be under active development.
Thanks!
from rltk.
Related Issues (20)
- Fix EBNF/selector interplay bug
- Move to LLVM 3.6 HOT 1
- Remove support for LLVM::JIT in anticipation of LLVM 3.6 release
- Rejecting a clause in a production HOT 2
- A more complete example for error production HOT 2
- require 'rltk/ast' pollute main Object with #match method, leaking it in a global scope HOT 7
- RLTK::LexingError carries a wrong line_offset. HOT 6
- :pointer argument is not a valid pointer
- Calling #type on RLTK::CG::Value throws an error
- JITCompiler segfaults on OSX
- Split the LLVM bindings into their own project: RCGTK
- RLTK::Parser.finalize use: fails to load cache on Windows
- left/right associativity maintains separate precedence stacks
- LLVM HOT 2
- undefined method `subclass_of?' HOT 1
- Requires Ruby >= 2.4.0 due to Filigree HOT 1
- rltk (3.0.1) does not work with filigree (0.4.1) due to check_type signature
- Cannot Generate YARD documentation
- Make a release? HOT 2
- "String not in language" error HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rltk.