Comparison for nested type about arrow-rs HOT 3 CLOSED

Yes, DataFusion should decide what null semantics it wants / make this configurable, and implement it using make_comparator if appropriate

from arrow-rs.

I'm not sure we want to support this within these kernels, in particular the null behaviour is not well-defined. I added support to the comparator for this, which will allow systems like DataFusion to define an ordering for nulls based on a config setting, as Spark and Postgres use different ordering.

There is an example of this here - https://docs.rs/arrow-ord/latest/arrow_ord/ord/fn.make_comparator.html#postgres-compatible-nested-comparison

This will also make it more obvious that the nested comparison is not vectorised

Edit: Filed #5942 to make this more discoverable

from arrow-rs.

I'm not sure we want to support this within these kernels, in particular the null behaviour is not well-defined. I added support to the comparator for this, which will allow systems like DataFusion to define an ordering for nulls based on a config setting, as Spark and Postgres use different ordering.

There is an example of this here - https://docs.rs/arrow-ord/latest/arrow_ord/ord/fn.make_comparator.html#postgres-compatible-nested-comparison

This will also make it more obvious that the nested comparison is not vectorised

Edit: Filed #5942 to make this more discoverable

Are you suggesting that if datafusion requires nested comparison, we could implement them in datafusion with make_comparator like what I had done? Arrow-rs only support the make_comparator kernel for nested type?

https://github.com/apache/datafusion/blob/d32747d09add5e1c670aa32fbe3294ecee15e3b7/datafusion/physical-expr/src/expressions/datum.rs#L28-L58

This is the code that utilize compare_op, and nested type is missing

from arrow-rs.