Checkpointing causes dataChange=False about delta-rs HOT 5 CLOSED

Looking at the checkpoint code, I see this line:

Am I misunderstanding what checkpoint does? Why is it being set to False?

from delta-rs.

@echai58 change seems to be done 3 years ago ^^, but reading the protocol it mentions data_change=false if files were already present in a table, so I guess that's why in this context.

You could check what the behavior is of delta-spark and if you see the data_change being preserved as-is?

from delta-rs.

@echai58 change seems to be done 3 years ago ^^, but reading the protocol it mentions data_change=false if files were already present in a table, so I guess that's why in this context.

You could check what the behavior is of delta-spark and if you see the data_change being preserved as-is?

Yeah, in my eyes a checkpoint isn't really a transaction because it doesn't generate a commit file, so i feel it should preserve it. But yeah, I'll look into what the delta-spark behavior is, good call.

from delta-rs.

@rtyler @roeap do you know why it's set to False perhaps?

from delta-rs.

@ion-elgreco Seems like checkpointing with delta-spark also sets dataChange to False. 🤷‍♂️ I guess that's the desired behavior.

I brought this up because I have a use case where I'm interested in figuring out which partitions were edited in the past n days, so I know which partitions to selectively z-order, so I don't have to z-order the entire table. I was relying on the add_actions table with the dataChange and modificationTime columns to determine this, but because of checkpointing, this seems to not work - do you have any alternatives?

from delta-rs.