willscott / carbs Goto Github PK

Currently, carbs is structured to first take a reader to a full car file, and generate an index for it, and then in a second phase support efficient lookups for cids within the car.

There are use cases where the car file is streamed in, but there's a desire to retrieve items before the car file is fully written.

• split Get from index generation interface
• create a streaming index that uses insertion sort to keep the index sorted as blocks are inserted
• add WriteCarbsWithWalker(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, destinationCarPath string, walk car.WalkFunc) (Carbs, error)

There's also potentially a useful higher-level exposed interface to support:

• add WriteCarbs(ctx context.Context, roots []cid.Cid, destinationCarPath string, walk car.WalkFunc) (Carbs, error)
  which accepts blocks to be written to the car file via ds.Put until the context is completed, and is responsible for re-ordering blocks per the walk function to create the canonical car.

would be nice