Do partial cluster decompression about libzim HOT 2 CLOSED

A solution may be to cache the partially decompressed cluster together with the current state of the decompression. Then, if/when a blob from the not-yet-decompressed portion of the cluster is later needed, decompression can be performed incrementally from the point at which it was previously suspended.

It is a tricky thing to do (while possible) :

• We don't want to invalidate the cache. So we must not reallocate the already decompressed data. It means that we must do a partial decompression to read the offsets and be able to get the total size, allocate a buffer for this size and then do another partial decompression (until the article to read) in this buffer. It would totally interleave the functionality of decompressing the content and reading it (where it is two different paths now)
• We have to check every time we want to access a blob in the cluster that the data has been actually uncompressed. Every access to the data must lead to a decompression and a modification of the buffer. It means that this should be thread safe as several threads may access the cluster in the same time.
• Storing the decompression state means storing the decompression stream (lzma_stream, z_stream or ZSTD_[CD]Stream) and maybe the compressed data (or we may have to do some complex synchronization to load the data later (to investigate)).

This is not a easy task. And I'm not sure we need it. At least we should do some measurement to be sure it worth it (especially with the new zstd algorithm that should speedup the decompression).

from libzim.

How is this going to play with caching of (decompressed) clusters? If the chances are high that other blobs will be used from the same cluster while it is in the cache, decompressing the entire cluster once is better than partially decompressing it multiple times. A solution may be to cache the partially decompressed cluster together with the current state of the decompression. Then, if/when a blob from the not-yet-decompressed portion of the cluster is later needed, decompression can be performed incrementally from the point at which it was previously suspended.

from libzim.