Discussion: make it easier for specify SQL --> function translation about arrow-datafusion HOT 7 CLOSED

@jayzhan211 asks #10374 (comment)

@alamb Do you think we should also rewrite the array operator to function in parser step? It is currently in optimizer step. I think the downside of moving array rewrite in parser step is that if any user expects different array function with the same syntax, then they can't do it since we don't have "user-defined" parser mechanism now. But the benefit is that we can eliminate intermediate binary expression.

The array operator to function is syntax like array1 || array2 -> array_concat, which is in ArrayFunctionRewriter now, so I'm thinking about whether we should move this to the parser or not.

I personally think moving the translation of array1 || array --> array_concat to the parser is a better idea as it will be more efficient than trying to rewrite an expr after the fact

from arrow-datafusion.

I rethought the issue in #10102, and I found it is strongly related to the idea of the user-defined parser mentioned here, that we can define the returned Expr given the registered function.

from arrow-datafusion.

#11180 -- looking very nice

from arrow-datafusion.

In general, I think the idea of allowing users to customize the behavior of the sql planner is reasonable. However I am not entirely sure if we need to modify the planner itself, or if it would be a better approach for users to implement rewrite passes after the existing planner runs 🤔

from arrow-datafusion.

For anyone following along, @samuelcolvin @jayzhan211 and myself are discussing proposals for this API in

#11137 and #11168

from arrow-datafusion.

To rewrite with sql planner

• date_part
• create_struct
• create_named_struct
• sql_overlay_to_expr
• sql_position_to_expr
• sql_substring_to_expr
• sql_compound_identifier_to_expr

from arrow-datafusion.

I filed #11207 to track the work to move the remaining functions to the user defined extension planner so closing this one

from arrow-datafusion.