Get statistics metadata about delta-rs HOT 4 CLOSED

maybe you are looking for get_add_actions
https://delta-io.github.io/delta-rs/usage/examining-table/

from delta-rs.

I think what you are looking for is under DeltaTable()._table.dataset_partitions(). We use that to construct the stats for each fragment for the pyarrow dataset

from delta-rs.

Oh cool. Thanks for the pointer. That gets me closer. Here's what I'm getting

import deltalake
t = deltalake.DeltaTable("mytable")
filename, info = t._table.dataset_partitions(t.schema().to_pyarrow())[0]
info

<pyarrow.compute.Expression ((((((((((((((((((date == 2024-01-03) and is_valid(username)) and is_valid(repo)) and is_valid(sha)) and is_valid(message)) and is_valid(created_at)) and (username >= "000600")) and (repo >= "0-vortex/0-vortex")) and (sha >= "00002c6c6353df7b4429ea6f4ca8f7674df6e8a0")) and (message >= "")) and (created_at >= 2024-01-03 04:00:00.000000)) and (date >= null[date32[day]])) and (username <= "zzzzseong")) and (repo <= "zzzzseong/algorithm")) and (sha <= "ffff8a90f31cba8f163fdd723c3451a04a8c39a2")) and (message <= "🩺 Checked PDS Health")) and (created_at <= 2024-01-03 05:59:59.000000)) and (date <= null[date32[day]]))>

So clearly the stuff I want is in there, however I'm not sure that it's actually programatically accessible from Python. I guess I could raise this upstream with PyArrow to ask for compute expressions to be more introspectable, but this seems like the wrong path.

Any further thoughts, aside from parsing the string repr?

from delta-rs.

Oh cool. Yes, that seems like it likely has the information that I'm looking for. Thank you @sherlockbeard !

from delta-rs.