`tree-sitter` grammar about djot HOT 11 OPEN

I'd like to point out that this treesitter repository for commonmark is now recommended for use over the repository linked before.

from djot.

This would be 100% accurate (but not incremental, which might be an issue on large files).

One of the reasons I'm really interested in djot is because editing CommonMark documents becomes a pain when the documents become fairly large. It'd be awesome if editing djot documents doesn't feel that way as well. 😅

from djot.

I don't know anything about tree-sitter, but this would be a great project for someone who does.

from djot.

Made a start on this in the tree-sitter directory.

from djot.

OK, it looks like we'd need to define some external scanners to make this work, since the regexes tree-sitter allows don't include any assertions. (We won't be able to distinguish open- and close-emph delimiters, for example.) But tree-sitter does allow you to define external scanners in C, so maybe we could do this.

from djot.

https://tree-sitter.github.io/tree-sitter/creating-parsers#external-scanners

from djot.

Oh, thanks, that one is much easier to understand!

from djot.

An interesting option for nvim would be to use the djot lua library itself to do the lexing, and call https://neovim.io/doc/user/api.html#nvim_buf_add_highlight()
directly to add the highlighting. (nvim has a Lua api.)
This would be 100% accurate (but not incremental, which might be an issue on large files).

from djot.

That function wants to have (line, column) locations, but column is byte indexed. We could convert our current byte offsets to (line, column-byte) pairs by counting newlines. Perhaps we should change the Lua library so that it can efficiently produce this output as well as absolute byte offsets.

from djot.

Interesting. Well, we could try the thing I suggest above and see how it works. The parser is very fast, so maybe it will be okay. It would require a few changes in the Lua library, but then it should be fairly straightforward.

from djot.

While using the lua engine directly might be a good solution for highlighting, having a treesitter grammar has other advantages. In neovim, treesitter is also used for editing (selecting, deleting etc.) and navigation. Basically, as a common language abstraction, it makes integration between arbitrary plugins and arbitrary languages easier, resulting in a more homogeneous/higher level experience. So that would be a pro for a treesitter solution. Also, treesitter is supported by many editors (emerging ones like helix and zed), while a lua solution would need some integration code for each editor. (Pretty excited by the future of djot btw.)

from djot.