Allow use of external dictionary about density HOT 7 CLOSED

Just to check we're talking about the same thing : density adds a 16 bytes header per "file" I suppose that's what you meant ?
Because the internal block structure is something else.

from density.

That would be taken care of by just passing DENSITY_ENCODE_OUTPUT_TYPE_WITHOUT_HEADER_NOR_FOOTER, right? That's definitely a start, but if there is a way to squeeze a few extra bytes out of the block, too (beyond DENSITY_BLOCK_TYPE_NO_HASHSUM_INTEGRITY_CHECK), that would be great. This wouldn't have to work with the stream API (so no need to support multiple blocks), and could assume detailed a priori knowledge of how to decode. Ideally, this would mean no header or footer at either the block or file level. For example, with liblzma from xz-utils, you can use raw lzma/lzma2 but you must pass the same options (dictionary size, literal context bits, literal position bits, and position bits) to the encoder and decoder manually.

If it's a lot of work you may want to just close this and mark it as wontfix—it is just another use case I thought you might be interested in supporting. I haven't done any research yet comparing density's performance on small (20-2000 bytes or so) buffers, it's quite possible density isn't well suited to this at all. If you want I can try to put together a benchmark to see how density performs in that area before you invest time working on this.

from density.

Well it depends, as of now there is no more pre-calculated default dictionary so very small datasets don't compress well because the algorithms don't have enough time to "learn".
In the case of your database you know what type of data is stored in the tables, so it would be very easy to create a pre-calculated dictionary for that type of data and density's speed / compression ratio would then be excellent on the smaller datasets.
Creating one's default dictionary and being able to send it to the API is something I've thought about and will implement in the not so distant future. It will give great flexibility and performance to the small file user by giving the algorithms a way to "fast-forward" their learning process.

from density.

Okay, that's definitely in the spirit of what I'm interested in. Is there any other data that be moved out of the block level data? I'd like to move out everything which is replicated for every compressed blob of data. Even a byte or two is significant when multiplied by a large number of entries.

from density.

No as long as you disable the hash sum integrity check, as of today's version the block structure overhead will be near zero.
There remains the block efficiency checks (2 bytes per block, a block being somewhere between 256K to 512K long), but for an efficiency check to be made a lot of data needs to be processed first, depending on the algorithm, but always more than 2K so for small datasets it is never made anyways.
To summarize, if you use DENSITY_ENCODE_OUTPUT_TYPE_WITHOUT_HEADER_NOR_FOOTER and DENSITY_BLOCK_TYPE_NO_HASHSUM_INTEGRITY_CHECK, as of today there is a near zero overhead for structures (the remaining few bytes being the efficiency checks if data is long enough).

from density.

Okay. I'll leave this open to track the separate dictionary stuff.

from density.

Available in 0.14.0

from density.