Alternative Storage backends? about delta-lake-internals HOT 4 CLOSED

Thanks for the feedback, I will reach out to them!

from delta-lake-internals.

Hey Jay,

if there is any viable (existing) alternative to HDFS that could be used that complies with the Delta storage requirements.

I haven't heard of any myself either but I could ask around and see if I can find anything. Can you point me at the Delta storage requirements and what alternatives you're interested in?

from delta-lake-internals.

Hey Jacek,

we are open to what we use, however, we try to come up with a solution that minimizes the effort in regards to setup & maintenance.

Our setup is Kubernetes based thus what we access as storage should be Kubernetes managed (even in the case of HDFS).

The infrastructure we are going to use are multiple Nvidia DGX boxes that will act as the Kubernetes cluster that are connected via network, allowing to use the inbuilt NVMe storage as well as the available compute resources. Due to this set up we were considering MinIO or potentially CephFS as alternatives to HDFS.

from delta-lake-internals.

I've got no experience with such a setup so I personally can't help much. I'd recommend posting this question at https://stackoverflow.com/questions/tagged/delta-lake to reach out to the Delta Lake community.

from delta-lake-internals.