Unexpected high costs on Google Cloud Storage about delta-rs HOT 5 CLOSED

I fixed this by adding a regular checkpoint creation function. This reduces the number of file operations.

PR around auto checkpoints is #913

@scheduler_fn.on_schedule(schedule="*/5 * * * *", memory=options.MemoryOption.GB_1 , timeout_sec=1000) def checkpointdb(request): dt = DeltaTable("gs://bucket/deltalake/feed" , storage_options={"google_service_account_key": json.dumps(google_service_account_key)}) dt.create_checkpoint() print(f"DB Checkpoint" , flush=True)

from delta-rs.

in order to write a delta table, we also need to always know the latest state of the table. as such ever read also requires us to read all relevant log files at least once. Usually there may be one or more list operations as well.

Are you creating checkpoints? if not, we have to read one commit file for at least very transaction that was created on a table, which can become very sizeable.

We have a PR in flight, that will allow us to be more economic in terms of reads, especially in append-only scenarios, where we can disregard a lot of the log - again, given there are checkpoints.

from delta-rs.

I'm going to close this, I don't believe there is something actionable for the delta-rs project here

from delta-rs.

I wouldn't have the slightest clue what class B operations even resemble, I don't use GCP myself.

If you can break it down into lingo to non-gcp users that would help

from delta-rs.

@ion-elgreco mainly class B operations are for reading objects from Google Cloud Storage.

@roeap I'm not sure about checkpoints. I haven't defined any myself, so if write_deltalake is not using them by default I would assume I was not using them. Based on the numbers I provided does it make sense to get so many read/list operations?

Note: I later changed the implementation of simply adding new parquets as I figured out I don't really need the functionality of delta lake. I just wanted to point it out if anyone else had a similar problem. Especially since this can incur high unexpected costs on cloud providers.

from delta-rs.