re::engine::RE2 ignores the target string utf8 flag about re-engine-re2 HOT 3 OPEN

likely related to #8 (comment) which has our incomplete fix for non-ascii matching. In general we always utf8::downgrade strings because native utf8 perl matching+capture can be horrendously slow - which is one reason why we switched to RE2 in a few performance critical areas

from re-engine-re2.

The fact one person wants an upgrade and one a downgrade is why this is tricky to fix ;-)

Ideally what needs to happen is re::e::RE2 needs to compile two RE2s, one as UTF-8 and one as Latin1, then pick (maybe one lazily, it needs to do at least one to handle errors). The first problem is however that's done has the potential to seriously affect performance for many cases.

There's also the issue it isn't as simple as you may think, because you can't even compile some regexps as Latin1 with re::engine::RE2, there's a 12 year old note in the todo that's still relevant: https://github.com/dgl/re-engine-RE2/blame/master/TODO

$ perl -Mutf8 -Mblib -e 'use re::engine::RE2 -strict => 1; /😀\x{1F01}/'
$ perl -Mutf8 -Mblib -e 'use re::engine::RE2 -strict => 1; /\x{1F01}/'
invalid escape sequence: \x{1F0 at -e line 1.

So maybe the overall approach is compile as UTF-8, then if needed lazily try Latin1. Maybe we could use the Perl regexp /a flag to actually mean Latin1, although that needs some more design thought; certainly how to do this while making it more perl compatible without regressing people's potentially performance critical code.

from re-engine-re2.

The workaround we actually use is to compile two patterns, one utf8 and one non-utf8, and then use the appropriate one for the target string. Successfully downgraded target strings get the non-utf8 pattern. That way we get the benefit of faster processing on target strings that don't need utf8 matching. It works for us, but that's because we don't need to use \x escapes above \xFF in our patterns.

from re-engine-re2.