New bucket serialization format about cacache-rs HOT 2 CLOSED

I'm curious if you plan to keep the whole "nested shasum parts" thing. I noticed that the JS cacache spends a fair bit of FS ops on that, and it seems like most file systems in use today can handle bazillions of files in a single dir.

On my machine, I'm seeing about a 2-5% performance boost in all the benchmarks that are sensitive to bucket loading efficiency when I make it just use a single layer of files in one big index folder. Not revolutionary, but not nothing. I haven't yet run it through a benchmark that creates millions of buckets, so it's possible that it's A Bad Idea, of course :)

from cacache-rs.

I did a spike towards this and realized this is way more trouble than it's worth -- generally, if you want performance, cacache::read_hash(_sync) is the way to go, as it's just as fast as a regular filesystem read. Considering the general slowness is pretty much I/O bound, with only a bit of JSON parsing delay (and serde_json is very fast)... I think I'm just gonna go ahead and say this wouldn't be worth the trouble just to squeeze another couple of percentage points in.

from cacache-rs.