Datafusion integration assumes table's data files are local about delta-rs HOT 10 CLOSED

@gopik it will be part of datafusion, see apache/arrow-datafusion#907

from delta-rs.

Yeah, unfortunately, datafusion uses arrow parquet readers, which only supports local file at the moment: https://github.com/apache/arrow/blob/master/rust/datafusion/src/physical_plan/parquet.rs#L181. I think this is best handled by the rust parquet reader with minor adjustments to datafusion's execution plan after that.

@nevi-me has plans to add S3 support to the parquet reader. If you are interested in extending the reader to support S3 or other cloud storages, I would recommend collaborating with him :)

from delta-rs.

I think this is best handled by the rust parquet reader with minor adjustments to datafusion's execution plan after that.

Makes sense to me!

@nevi-me has plans to add S3 support to the parquet reader. If you are interested in extending the reader to support S3 or other cloud storages, I would recommend collaborating with him :)

Sounds good, I'll keep an eye on it and try and contribute an Azure reader when the time comes.

from delta-rs.

What could work in the interim is to use DataFusion's in-memory datasource (https://docs.rs/datafusion/2.0.0/datafusion/datasource/memory/index.html). When we have async-support on Parquet, then we can change to the relevant methods.

from delta-rs.

@nevi-me is there a bug anywhere to track S3 support? I took a brief look in the Arrow and Datafusion repos and didn't find anything. If you're open to it it's something that we could potentially look in to contributing.

from delta-rs.

@meastham feel free to start a discussion for s3 support in the upstream datafusion github repo or in the arrow dev mailing list.

from delta-rs.

Given object store support in datafusion, can a blob path integration be implemented assuming we have appropriate blobstore implementation of object_store interface?
https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/datasource/object_store/mod.rs

I understand that given this, we can pass the file names prefixed with appropriate storage handler name from delta-rs, but my question is, is datafusion execution plan integration with this data source complete or is it still in progress?

from delta-rs.

@gopik yes, we are pending on upstream object store support for s3. datafusion execution plan integration is all complete other than partition column support, which should be fairly straight forward to add.

from delta-rs.

@houqp When you say upstream object support for s3, will that be part of datafusion project or it'll be part of an integration that is embedding datafusion?

from delta-rs.

With the adoption of object_store, the datafusion integration now supports all storage backends - there are integration tests as well :).

https://github.com/delta-io/delta-rs/blob/main/rust/tests/integration_datafusion.rs

from delta-rs.