Comments (5)
Traditionally, specifying new input has never reset anything in the lexer. This is to support use cases where input might be assembled in chunks or from different sources, but made to appear as if it were a continuous source of tokens to the parser. Honestly, I'm not sure how well this is defined though. So, would need to look at it further.
from ply.
This is to support use cases where input might be assembled in chunks or from different sources
How would this work at yacc.LRParser
level ? As far as I can see, current API will loop until get_token
returns a false value (ie, None
), at which point it will insert a $end
token.
Background: I tried to use ply to represent the grammar of a protocol, which my program analyses as the protocol is being used between two independent entities. So my lexer is very custom. As I want parser's output to be updated as soon as possible after link activity, I want to push tokens as soon as they are received - without checking if they make the grammar reach topmost rule (where no state is lost if $end
is encountered) as this would defeat the purpose of using ply automated state machine generation. So far, the only solution I found was to extend yacc.LRParser
so as to reuse its internals - so I would be very interested to know of a better way if it exists.
from ply.
Well, supporting the use case you are describing (supply input in multiple chunks) would be pretty useful, and is possible with traditional bison/flex.
However when I opened this ticket I came to the conclusion that this was not supported (I cant remember the exact details but for sure I read the source), thats why I asked.
Anyway, my code now reuses the parser instance and calls input, but I could instantiate a new parser instead.
So on my part, input() which allows multiple chunks would be the best option.
from ply.
An incremental lexer would require passing a modified lexer and/or tokenfunc to the parser. For example:
def modified_get_token():
# Return the next token
...
return tok
yacc.parse(tokenfunc=modified_get_token)
Within the modified_get_token() function, you could drive the lexer, check for EOF, and feed input in a more customized manner.
Again, I don't know if this has ever been fully explored in PLY. Maybe needs some more documentation. I will look into this.
from ply.
For now, I'm going to leave the parsing state alone on new input. The lexer() can have its state manually reset on new input if needed. There are more advanced uses where you would not want to reset the state on input.
from ply.
Related Issues (20)
- Warn user about tuple flattening in precedence table HOT 3
- PyPI package HOT 2
- Trying to get in touch regarding a security issue
- "global flags not at the start of the expression" error when using in-line global regex flag with Python 11+ HOT 1
- Please make a new release HOT 1
- PLY not compatible with python 3.12 HOT 1
- extreme slowness HOT 5
- Read grammar from files HOT 2
- Re examples/ansic HOT 1
- lextab.py and yacctab.py HOT 13
- Exporting gramma for documentation
- ssl cert issue for the website
- License HOT 2
- "state" is ambiguous in the documentation. HOT 1
- lex.lex() can get some rules in the wrong order. HOT 1
- Documentation in doc/ply.md lacks important details about creating a custom lexer. HOT 6
- Import several module with lex()
- allow user actions in debug mode HOT 1
- Add MIT License
- PLY VHDL parser not producing a parser tree... HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ply.