Add push down sort to the source (table provider) about arrow-datafusion HOT 4 OPEN

Possibly related: #7871

from arrow-datafusion.

Possibly related: #7871

@alamb Thank you for the reply!

I've read discussion in #7871 and think that this case is different.

I don't want to say that MySourceExec can do sort better than DF does.
I'd like to tell to MySourceExec how it should be load an external batches of data: from the beginning or from the tail (in reverse order).

How can I do that without knowing about sorting?

from arrow-datafusion.

It the example TableProvider may know that it needed to provide only last record batch (or the latest parquet file from folder).

The provider can tell DataFusion it produces data in some pre-existing order via
https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#tymethod.properties
which can then communicate the sortedness via https://docs.rs/datafusion/latest/datafusion/physical_plan/struct.PlanProperties.html#method.output_ordering

However, that usecase is for data that is already always sorted. It doesn't permit DataFusion to say "I won't use the sort order you reported so you don't have to honor the contract" 🤔

from arrow-datafusion.

(BTW @NGA-TRAN and I worked on a very similar feature in InfluxDB IOx -- and we implemented a special operator that knows how to do this "read only the most recent file" for queries very much like your example above

SELECT * FROM Events
ORDER BE event_time DESC
LIMIT 1000

There is additional discussion / links on #10313

I think we would be interested in collaborating / pushing some of this logic upstream to datafusion. @matthewmturner and @suremarc may also be interested

from arrow-datafusion.