I'd like to come up with a proper highlighting algorithm to use with QScintilla and I need a little bit of help/advice. In QScintilla basically you apply styles to chunks of texts using byte positions, here you can find a perfect explanation of how it works.
Right now I've coded a little snippet to understand the basics of tree-sitter... the goal would be apply a monokai style using the existing python grammar, consider this:
import textwrap
from tree_sitter import Language, Parser
def print_node(node):
pos_point = f"[{node.start_point},{node.end_point}]"
pos_byte = f"({node.start_byte},{node.end_byte})"
print(
f"{repr(node.type):<25}{'is_named' if node.is_named else '-':<20}"
f"{pos_point:<30}{pos_byte}"
)
def traverse(tree):
def _traverse(node):
print_node(node)
for child in node.children:
_traverse(child)
_traverse(tree.root_node)
if __name__ == '__main__':
code = textwrap.dedent("""\
# こんにちは
def hello():
if world:
foo()"""
)
print(repr(code), len(code.encode("utf-8")))
print('-'*80)
for l in code.splitlines(True):
print(repr(l), len(l.encode("utf-8")))
print('-'*80)
grammar_name = "python"
parser = Parser()
parser.set_language(PY_LANGUAGE) # <-- IMPORTANT: Set PY_LANGUAGE to use your shared library
tree = parser.parse(bytes(code, "utf8"))
traverse(tree)
'# こんにちは\ndef hello():\n if world:\n foo()' 58
--------------------------------------------------------------------------------
'# こんにちは\n' 18
'def hello():\n' 13
' if world:\n' 14
' foo()' 13
--------------------------------------------------------------------------------
'module' is_named [(0, 0),(3, 13)] (0,58)
'comment' is_named [(0, 0),(0, 17)] (0,17)
'function_definition' is_named [(1, 0),(3, 13)] (18,58)
'def' - [(1, 0),(1, 3)] (18,21)
'identifier' is_named [(1, 4),(1, 9)] (22,27)
'parameters' is_named [(1, 9),(1, 11)] (27,29)
'(' - [(1, 9),(1, 10)] (27,28)
')' - [(1, 10),(1, 11)] (28,29)
':' - [(1, 11),(1, 12)] (29,30)
'if_statement' is_named [(2, 4),(3, 13)] (35,58)
'if' - [(2, 4),(2, 6)] (35,37)
'identifier' is_named [(2, 7),(2, 12)] (38,43)
':' - [(2, 12),(2, 13)] (43,44)
'expression_statement' is_named [(3, 8),(3, 13)] (53,58)
'call' is_named [(3, 8),(3, 13)] (53,58)
'identifier' is_named [(3, 8),(3, 11)] (53,56)
'argument_list' is_named [(3, 11),(3, 13)] (56,58)
'(' - [(3, 11),(3, 12)] (56,57)
')' - [(3, 12),(3, 13)] (57,58)
In that tree I see some missing information, how should I tweak my algorithm to print all relevant information I need to make syntax highlighting?
In any case, could you please explain the overall algorithm to use in combination with tree-sitter? In the past when using another libraries I've always found making syntax highlighting with large texts wasn't a trivial task... hopefully with tree-sitter this will be much easier :)
Thanks.
Ps. This is basically a question but at the end of this thread maybe we'll get some useful explanations that you can use to add to the docs... also, once I've got a QScintilla example working I could make a PR to add it as example/test if you want.