Use tokenized sequences when comparing edit scripts about logifix HOT 2 CLOSED

In favor of spatial locality we should probably change the signature of the tokenizer to take std::string_view while we're at it.

Re2c by default checks for \0 as a sentinel, string_views obviously have no sentinels, but I think there is a simple workaround.

from logifix.

If we change the signature of the tokenizer to tokenize(std::string_view str) -> std::vector<std::pair<sjp::token,std::string_view>> we should be able to just do:

- bool edit_scripts_are_equal(std::string o, std::string a, std::string b, std::string c) { 
+ bool edit_scripts_are_equal(std::string_view o, std::string_view a, std::string_view b, std::string_view c) { 
...
-     auto diff = nway::diff(o, {a, b, c}); 
+     auto diff = nway::diff(tokenize(o), {tokenize(a), tokenize(b), tokenize(c)}); 

It is significant that sjp::token is first in the pair since == on pairs compare the left member side first.

from logifix.