What's the best highlighting algorithm I should use with py-tree-sitter? about py-tree-sitter HOT 8 CLOSED

For the general strategy, the configuration interface for Atom's Tree-sitter-based syntax highlighting is documented in the Atom book. The implementation is here. In addition, there is a completely separate Rust implementation that's a bit newer, but is not yet documented. This new implementation is what uses the highlights.css files in some of the Tree-sitter parser repositories.

Unfortunately, the best Tree-sitter API to use, the TreeCursor, is not currently exposed to Python in py-tree-sitter. The purpose of the TreeCursor is to let you walk the syntax tree efficiently, in a depth-first fashion, without recursion. Both of the highlighting systems that I mentioned above rely on the TreeCursor.

from py-tree-sitter.

Version 0.0.7 has the TreeCursor. I would suggest using an approach similar to what Atom does, where it uses the TreeCursor to walk the tree, and render highlights according to some mapping of Tree-sitter nodes to highlighting classes. CSS is a good language for defining this mapping: that's what I've been using so far. And there are two separate implementations of the CSS mapping: one in Atom, and one in Tree-sitter itself.

from py-tree-sitter.

Max, this is great news and I must say this new release is pure awesomeness, really exciting ;)

Tomorrow I'll be playing with 0.0.7 for sure... at first I was considering if it'd be worth factor out my highlighters+styles (right now i'm supporting syntect + pygments) so the editor would support pygments+syntect+tree-sitter but I can't think of any good reason for still supporting those regex-based engines, it adds over-complexity for no real reason.

At first I was scared tree-sitter wasn't supporting as many languages as pygments but then after checking how the grammar.js files look like I'm not scared anymore, so I think I'll throw-off to the bin the other highlighters and become a tree-sitter user :D

Let me tell you I think py-tree-sitter has the potential to become the "standard" python reference library for parsing/highlighting (right now i'd say pygments is the winner by far, isn't?)... IMHO one of the keys that will make that to happen is if the number of available tree-sitter grammars will grow consistently. I think one of the keys for this to happen is the file-format, at the end of the day new contributors will add stuff if that can be done easily ;) . Think of antlr, why did it success? Because creating grammars there only requires being familiar with EBNF and you'll get up and running...

Anyway, thank you very much, great job with 0.0.7! :D

from py-tree-sitter.

Issue-Label Bot is automatically applying the label question to this issue, with a confidence of 0.91. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

from py-tree-sitter.

Thanks, tomorrow I'll review all this info carefully, specially that Rust implementation may help... in the past I'd port syntect (rust library) to python so I've got a bit of experience wrapping rust code into python...

In any case, before using any of the complex efficient solution you've posted I'd like to implement the most naive way that at least allowed me to work with small files <100kb of python code in the editor, what'd you suggest?

DFS tree traversal in pure python would be that bad? Also, how can I print the missing information in my above snippet?

from py-tree-sitter.

I've got a bit of experience wrapping rust code into python...

Creating the Python bindings for TreeCursor might be familiar to you then! 😁

Otherwise, I'm sure it's possible to perform syntax highlighting with the existing APIs, but it's not something I've explored.

from py-tree-sitter.

Ok, I added bindings to the new query API. See the pattern-matching section of the README. There are queries for syntax highlighting checked into some of the language repos (Example).

There is still a little bit of work to do to get the python binding supporting all of the features of the query API. Specifically, the match? predicates (for doing regex matches on nodes' text) will not do anything right now. I'm not sure when I'll get around to this, but even without this, syntax-highlighting python should work alright.

from py-tree-sitter.

We now have match? predicates merged in HEAD, so I'm closing this. Let me know if you need anything more for this use case.

from py-tree-sitter.