Is pre-compile pattern string in regexp_match operation about arrow-datafusion HOT 3 OPEN

Hi @zhuliquan -- I believe you are correct that the regexp is re-compiled for each batch. This is mentioned as one of the use cases for #8051

I think it would be a nice improvement to look into pre-compiling the regexp somehow, though last time I checked the only benchmark we have (one of the clickbench queries) compiling the regexp was not a significant consumer of time

from arrow-datafusion.

Hi @zhuliquan -- I believe you are correct that the regexp is re-compiled for each batch. This is mentioned as one of the use cases for #8051

I think it would be a nice improvement to look into pre-compiling the regexp somehow, though last time I checked the only benchmark we have (one of the clickbench queries) compiling the regexp was not a significant consumer of time

I think should pre-compiled when create binary-physical-expr, instead of evaluating record_batch.

from arrow-datafusion.

I think should pre-compiled when create binary-physical-expr, instead of evaluating record_batch.

I think this makes sense to me. Thank you

One thing to look into might be to replace the regexp match operator with a function (and have the function's implementation store the precompiled regexp)

One tricky part might be serialization (in datafusion-proto). We can probably handle it via an extension codec or something.

from arrow-datafusion.