worst case compression wrongly evaluated about finitestateentropy HOT 6 CLOSED

Does this bug mean that FSE is not ready for use in a production environment? @Cyan4973 Do you think that 512 bytes should always be enough for the encoded bit stream, and if not, what it should be?

from finitestateentropy.

It's a bit early to call FSE "production ready", although that's the plan, and we are slowly getting there. My intention is to move away from "beta status" by the end of the year.

As a sidenote, FSE is getting some good testing through its usage within Zhuff, with several million files correctly compressed/decompressed so far.
(see : https://groups.google.com/forum/#!topic/lz4c/OmA_uSnaO3Q)

Do you think that 512 bytes should always be enough for the encoded bit stream, and if not, what it should be?

If you mean "enough for the header", yes that's what I initially thought.
Unfortunately, Artem tests prove that there is some kind of expansion I've not correctly estimated.
That's something I intent to sort out and correct within the month.

As a sidenote, the issue is only visible when compressing "very" large blocks (such as quite a few megabytes), which is not recommended : in most circumstances, it's much better to compress smaller blocks (such as, typically, 64KB), to ensure proper entropy local adaptation, hence better overall compression ratio.

from finitestateentropy.

Thank you, Yann, for your prompt response! This sounds great. Depending on how my compression experiments go this week, we may end up using FSE in our compressed mesh file format. Our largest blocks are 256 KiB. I'll let you know if we encounter any problems!

from finitestateentropy.

Latest beta solves this issue.
16fc2e3

Max inflation has been evaluated at 0.45%, plus header.
FSE_compressBound() is a bit of an over-evaluation currently, but it needs to be simple and fast, and it is safer for this function to over-evaluate a bit, rather than under-evaluate as it did up to now; resulting in memory errors.

from finitestateentropy.

Thank you!

from finitestateentropy.

Committed to master

from finitestateentropy.