dgl / re-engine-re2 Goto Github PK
View Code? Open in Web Editor NEWre::engine::RE2 - RE2 regexp engine interface for Perl
Home Page: http://search.cpan.org/dist/re-engine-RE2
re::engine::RE2 - RE2 regexp engine interface for Perl
Home Page: http://search.cpan.org/dist/re-engine-RE2
NAME re::engine::RE2 - RE2 regex engine SYNOPSIS use re::engine::RE2; if ("Hello, world" =~ /Hello, (world)/) { print "Greetings, $1!"; } DESCRIPTION This module replaces perl's regex engine in a given lexical scope with RE2. RE2 is a primarily DFA based regexp engine from Google that is very fast at matching large amounts of text. However it does not support look behind and some other Perl regular expression features. See RE2's website <http://code.google.com/p/re2> for more information. Fallback to normal Perl regexp is implemented by this module. If RE2 is unable to compile a regexp it will use Perl instead, therefore features not implemented by RE2 don't suddenly stop working, they will just use Perl's regexp implementation. METHODS To access extra functionality of RE2 methods can be called on a compiled regular expression (i.e. a "qr//"). * "possible_match_range([length = 10])" Returns an array of two strings: where the expression will start matching and just after where it will finish matching. See RE2's documentation on PossibleMatchRange for further details. Example: my($min, $max) = qr/^(a|b)/->possible_match_range; is $min, 'a'; is $max, 'c';' * "named_captures()" Returns a hash of the name captures and index. Example: my $named_captures = qr/(?P<a>\w+) (?P<d>\w+)/->named_captures; is $named_captures->{a}, 1; is $named_captures->{d}, 2; * "number_of_capture_groups()" Return number of capture groups Example: my $captures = qr/(Hello), (world)/->number_of_capture_groups; is $captures, 2; PRAGMA OPTIONS Various options can be set by providing options to the "use" line. These will be pragma scoped. * "-max_mem => 1<<24" Configure RE2's memory limit. * "-strict => 1" Be strict, i.e. don't allow regexps that are not supported by RE2. * "-longest_match => 1" Match on the longest match in alternations. For example with this option set matching "abc" against "(a|abc)" will match "abc", without depending on order. * "-never_nl => 1" Never match a newline ("\n") even if the provided regexp contains it. PERFORMANCE Performance is really the primary reason for using RE2, so here's some benchmarks. Like any benchmark take them with a pinch of salt. Simple matching my $foo = "foo bar baz"; $foo =~ /foo/; $foo =~ /foox/; On this very simple match RE2 is actually slower: Rate re2 re re2 674634/s -- -76% re 2765739/s 310% -- URL matching Matching "m{([a-zA-Z][a-zA-Z0-9]*)://([^ /]+)(/[^ ]*)?|([^ @]+)@([^ @]+)}" against a several KB file: Rate re re2 re 35.2/s -- -99% re2 2511/s 7037% -- Many alternatives Matching a string against a regexp with 17,576 alternatives ("aaa .. zzz"). This uses trie matching on Perl (obviously RE2 does similar by default). $ perl misc/altern.pl Rate re re2 re 52631/s -- -91% re2 554938/s 954% -- NOTES * No support for "m//x" The "/x" modifier is not supported. (There's no particular reason for this, just RE2 itself doesn't support it). Fallback to Perl regexp will happen automatically if "//x" is used. * "re2/dfa.cc:447: DFA out of memory: prog size xxx mem yyy" If you attempt to compile a really large regular expression you may get this error. RE2 has an internal limit on memory consumption for the DFA state tables. By default this is 8 MiB. If you need to increase this size then use the max_mem parameter: use re::engine::RE2 -max_mem => 8<<23; # 64MiB * How do I tell if RE2 will be used? See if your regexp is matching quickly or slowly ;). Alternatively normal OO concepts apply and you may examine the object returned by "qr//": use re::engine::RE2; ok qr/foo/->isa("re::engine::RE2"); # Perl Regexp used instead ok not qr/(?<=foo)bar/->isa("re::engine::RE2"); If you wish to force RE2, use the "-strict" option. BUGS Known issues: * Unicode handling Currently the Unicode handling of re::engine::RE2 does not fully match Perl's behaviour. The UTF-8 flag of the regexp currently determines how the string is matched. This is obviously broken, so will be fixed at some point. * Final newline matching differs to Perl "\n" =~ /$/ The above is true in Perl, false in RE2. To work around the issue you can write "\n?\z" when you mean Perl's "$". Please report bugs or provide patches at <https://github.com/dgl/re-engine-RE2>. AUTHORS David Leadbeater <dgl[at]dgl[dot]cx> COPYRIGHT Copyright 2010 David Leadbeater. Based on re::engine::PCRE: Copyright 2007 Ævar Arnfjörð Bjarmason. The original version was copyright 2006 Audrey Tang <[email protected]> and Yves Orton. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. (However the bundled copy of RE2 has a different copyright owner and is under a BSD-like license, see re2/LICENSE.)
I noticed that re-engine-RE2-0.17 includes re2/doc/xkcd.png file which comes from https://xkcd.com/208/ whose license is https://creativecommons.org/licenses/by-nc/2.5/ which prohibits a commercial use. That's prevents from redistributing re-engine-RE2-0.17 in Fedora Linux distribution.
Could you please remove that image from the next release, or negotiate a more free license https://docs.fedoraproject.org/en-US/legal/allowed-licenses/ with XKDC author?
Sample fail report: http://www.cpantesters.org/cpan/report/ee062c2a-0d9e-11ee-bf69-46044ed2e7f5
In the commit
commit c0bd4734ae7c7bb5045ad389161fac511cc8544d
Author: Yves Orton <[email protected]>
Date: Fri Jan 27 16:25:48 2023 +0800
fixup re::engine::RE2 for perl 5.37.8 and later (#10)
I found this piece:
+#if PERL_VERSION >= 37 && PERL_SUBVERSION >=7
+ rx->logical_nparens = rx->nparens;
+#endif
@demerphq , you may be interested, this does not cover 5.38, right?
If the pattern is such that it should successfully match the empty string, then matching against the pattern in list context with the global modifier set does not terminate. For example:
my $s = "tood";
my $p = "a*";
my @m = $s =~ /$p/g;
In Perl, the above yields an array of $n + 1
empty strings, where $n
is the length of $s
. When using RE2, the above does not terminate and causes the machine to run out of memory.
This is a known bug, but I wanted to document it here with a workaround.
re::engine::RE2, at least up through version 0.17, ignores the utf8 flag of the target string during regex matching, and instead operates on the target string's internal SV buffer contents as if it had the same utf8 flag value as the pattern.
This means, for example, that if the target string has the utf8 flag set, but the pattern does not have the utf8 flag, RE2 would treat the target as not having the utf8 flag set, which can prevent proper matching.
use re::engine::RE2;
print "match\n" if ("\x{2460}" =~ /\p{No}/);
In this example, the "Circled Digit One" character, \x{2460}
, should match the pattern looking for a character of category "Number, other", \p{No}
. It matches using Perl's built-in regex matching, but it does not match with RE2. RE2 is operating on the utf8-encoded bytes of the target string's internal buffer as if they were Latin-1 encoded bytes.
Also, if the target string does not have the utf8 flag set, but the pattern does, RE2 would treat the target as having the utf8 flag set, which can prevent proper matching.
use re::engine::RE2;
my $pattern = "\\p{Sm}\x{2461}?";
print "match\n" if ("\x{F7}" =~ /$pattern/);
The "Division sign" character \x{F7}
should match the pattern looking for a character of category "Symbol, math", \p{Sm}
. It matches using Perl's built-in regex matching, but it does not match with RE2. RE2 is operating on the Latin-1 encoded byte of the target string's internal buffer as if it were a utf8-encoded buffer.
As a further consequence of this bug, the utf8 flag is set incorrectly on capture buffers, using the flag value of the pattern instead of the flag of the target, which can corrupt the captured text.
use re::engine::RE2;
if ("\x{2460}" =~ /(.*)/) {
print "equal\n" if ("\x{2460}" eq $1);
}
In this example the captured string should be equal to the target string, but using RE2, they are not equal because it failed to correctly set the utf8 flag on the capture buffer. The utf8-encoded bytes of the target string's internal buffer were copied to the capture buffer as if they were Latin-1 encoded, corrupting the text.
In short, RE2 only works correctly when the utf8 flag value of the target string matches the utf8 flag value of the pattern string. Perl programs are not supposed to need to know utf8 flag values or manipulate them. But in this case, doing so is necessary to work around the RE2 bug. The simplest way to accomplish this is to utf8::upgrade the target string just before matching and utf8::upgrade the pattern string just before matching, or just before compiling in the case where the pattern is compiled. For example:
use re::engine::RE2;
my $target = "\x{2460}";
my $pattern = "(\\p{No})";
utf8::upgrade($target);
utf8::upgrade($pattern);
if ($target =~ /$pattern/) {
print "match\n";
print "equal\n" if ($target eq $1);
}
Using utf8::upgrade on a string is generally a safe operation. It should have no consequences outside of this RE2 bug, and it will be compatible with any future RE2 version that fixes this bug.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.