Add a method to get the cluster size of a ZIM file about libzim HOT 12 CLOSED

@mgautierfr Might be interesting as your are polishing zimmerge/zimpatch Kiran's code

from libzim.

@mgautierfr The problem here is that this is not written in the ZIM. Could we add it as a new header?

from libzim.

No :)

It would be a change (potentially incompatible) in the zim format.
And cluster is a concept internal to the zim format (and the libzim). A user code as no access to the cluster. There is no api to access it.
So, knowing its size is useless.

from libzim.

Yes, I ask for a new API. Considering that the Header can be extended without breaking the format, a default value shoudl just set.

from libzim.

There is no way to access cluster outside of libzim. And I want it to stay that way. (The same with dirent) Internal things must stay internal.

What is your use case ? How this would help you ?

from libzim.

@mgautierfr zimrecreate and zimpatch/zimdiff

from libzim.

How they could use it ?
zimrecreate do not need it at all.

from libzim.

These tools should should be able to know what is the ZIM cluster size to be able to then recreate one with the same clustersize.

from libzim.

If you provide the same content and compress the same way, the cluster should have the same size.

from libzim.

@mgautierfr How do I guaranty "the same way" if I ignore the cluster size? Considering the cluster is a variable in the compression process.

from libzim.

I cannot tell you how to avoid to use the cluster size if you don't tell me how you would use it.

Honestly, I don't think is is possible to zimpatch to guaranty this "same way" using libzim. I've already change the way zim's creator decides (and when) to close a cluster and this is totally internal. There is no way to force a specific set of articles to go in the same cluster.
We have one naive heuristic to determine which articles go in which cluster but this may change in the future. We may want to regroup articles per mimetype (because compressing content of the same type may be more efficient), or try different order of the article and take the best compression ratio, or regroup css and js together because there is a lot of chance that they are shared by all articles, or....

We cannot guaranty that articles will be written in the same clusters the same way the x years old implementation was doing it.
And I don't want to guaranty that.

from libzim.

We don't have the information about a cluster size (except by decompressing it and see how much have been read).
libzim, even internally, never try to get the size of a cluster. So we will not provide a public API to get it.

from libzim.