directory content is only implicitly known about borg HOT 6 OPEN

Can we efficiently access random items in metadata[b'items']? The code accessing that so far is strictly linear access.

If we could, a directory tree index could be added separately, only containing indexes into the items. This could speed up all content-reading actions significantly if only a subset is accessed.

Edit: Ah, no, msgpack only has a streaming unpacker it seems. But maybe we can seek before feeding data to unpacker and this way emulate random access?

from borg.

basically to make this safe, there is need for dependency tracking, so metadata of a directory will only get written after its content is done

this could be tracked as tree of contents that constructs as unpacking is done, and deconstructs as items are written,

every time a treee turns empty, we can write directory metadata

the pipeline can be created in a way that a tree cannot turn empty unless all its content is processed

from borg.

That would make it unnecessary to make another pass over all dirs in borg extract, but doesn't speed up "random access" to directories / files for listing/extracting subtrees.

Concrete example: listing all files in /home takes a minute or so, listing a subtree with a few dozen files still takes ~20 s (because iter_items / unpack_many needs to read, unpack, decode, filter all items)

from borg.

well, a backup tool has to make some trade-offs, - since the metadata is supposed to be sorted, i believe that binary search is reasonably fast for that kind of access

from borg.

If I understand things correctly accessing items non-linearly is not trivial to do.

from borg.

(I'm not saying that this is an actual problem that needs solving, I think it's "fast enough", at least for me)

from borg.