jankaul / iceberg-rust Goto Github PK

It looks like this project does have the capability to insert and update records in an existing iceberg table. Am i correct this? Look forward to hearing!

After ccbfae6, the append api in transcation become pub fn append(mut self, files: Vec<(String, FileMetaData)>).

Some unit test seems not modify accroding it.

The current design of the catalog trait doesn't allow the use of the REST catalog. The catalog trait has to be changed.

I see that the repo name has changed to iceberg-rust_archive and I saw a tweet about the following new repo https://github.com/apache/iceberg-rust .

@JanKaul given that you are a contributor on that repo, is it safe to say that it's been migrated to there?

Perhaps you can just put a note and a link to the new repo in the README to direct anyone that stumbles onto this.

Thanks

Hi,

First of all thank you so much for carrying the torch for bringing Iceberg support to Rust!

I've been looking for a simple to deploy Iceberg project that I can just run locally using Docker. Dremio-oss comes close but I'm having an issue with it and in any case, I'm still confused about the Iceberg Catalog.

Long story short, I'm wondering whether we can wrap up what you've done so far in a Docker image, expose some of the functionality in a CLI and run a small webserver to expose the REST API. I'm no Rust expert but I might be able to handle the CLI and Docker parts as I've done that before (https://github.com/prql/prql-query/).

Even if the functionality is currently not complete, my sense is that having something that's easy to deploy and try out might get more people to play with it who might then be inspired to implement missing functionality that they would like to see.

My personal requirements are quite limited as I have folders of parquet files in Minio buckets that I would just like to add to an Iceberg manifest and expose as an Iceberg table. My data arrives in whole files at a time so I have no need for updates or insert so I got the impression that the current state of the project might already be able to handle that use case.

WDYT? I'm eager to help with the Docker part, my biggest limitation is just time. If you could just provide some guidance on the traits and functions that you've implemented then I'll see what I can do.

Hi there, I'm building something that uses datafusion for parts of the planning and would like to be able to write results directly to Iceberg. Is the intent of this project to support that eventually? How much effort would it take and could I potentially help with that?

Hi there - I think what you are working on here is great. Im just curious if theres a reason for not contributing to what i believe is the official rust implementation. no judgement im just trying to get a lay of the land as im evaluating the options if i wanted to start contributing to rust / datafusion implementations for iceberg.

Hey there! I've been playing around with your lib, nice work 💪. I am curious though, have you put this down in favor of https://github.com/apache/iceberg-rust ? This project seems a bit ahead of that one, but has less activity.

Currently the all files are witten to the same directory even if partitioning is used. The files belonging to different partitions should be written to different directories.

ExprRewriter