"unexpected EOF" from some salza2 deflate data about chipz HOT 2 OPEN

smaller/non-random version

(let* ((d (coerce #(161 16 164 216 206 26 13 97 172 65 152 52 116 77 19 10) '(simple-array (unsigned-byte 8) 1)))
       (v (salza2:compress-data d 'salza2:deflate-compressor)))
  (with-simple-restart (continue "continue")
    (let ((x (time (chipz:decompress nil 'chipz:deflate v))))
      (assert (equalp d x)))))

from chipz.

This is an issue with chipz. decode-value ensures as many bits as are needed for the longest code, which can read bits past the end of the input if the code for the end-of-block marker isn't at least as long as all other codes. An encoder will usually not produce such a tree since EOB is by definition the least frequently used symbol - except when the last block uses the fixed Huffman code. Of course, a stupid/malicious encoder could also trigger this in a dynamic block.

Since this issue only affects the rare raw Deflate (zlib & gzip have trailing checksums that end up protecting against this) and repeatedly calling ensure-bits in the hottest part of the code is probably a bad idea, it might be worth it to write a separate version of decode-value for raw Deflate.

from chipz.