Read persistent data in reverse order about jylis HOT 2 OPEN

Both of those options sound great. It also doesn't sound like they're mutually exclusive and at some point both features could exist.

from jylis.

So, for context, persistence is currently done with append-only log files, where all operations that mutate state get their delta-state CRDT added to a log for that keyspace. This is very similar in concept to how Redis AOF files work (see Redis docs on persistence for more info). So, in other words, the data becomes available in the same it became available when first being written, following each of the same operations that were observed. Note that this may be different in practice from the order of timestamps in your TLOG instance.

The append-only format would make it difficult to write the data in reverse order (it would become prepend-only at that point, which wouldn't have the same desirable characteristics in terms of dealing with file handles). It would also be fairly costly to try to read the data in reverse order, and doing so would significantly increase the time it takes to load the data from disk in general.

I think there are two general ways of alleviating this pain point, which would be better options:

• 1️⃣ Adding some kind of new SYSTEM command that lets you check when the disk data is fully loaded.

  • This could then be observed by the application when managing the pool of cluster connections, to avoid sending other commands to an uninitialized node.
  • Doing this kind of model would also assist with other potential issues of directing production traffic to a node that is still initializing, like the fact that such traffic might have longer latencies while competing with the disk restore work.

• 2️⃣ Periodic compaction of the append-only log files, which would have a few desirable effects:

  • Data after compaction would take up an amount of space on disk that is proportional to the amount of data in the database, instead of taking up a grow-only amount of space on disk with lots of additional overhead from storing the individual (sometimes overlapping) deltas.
  • The overall time to load the data back into the database would be significantly faster.
  • Data from a single large TLOG instance would be loaded all-at-once instead of incrementally, avoiding the original problem you talk about with not all data being available yet.

from jylis.