Malloc size error in chmd.c:1327:34 about libmspack HOT 2 CLOSED

Thank you for the report.

What happens here is that the file size (u64) is cast to an int (i32), and then cast to size_t (u32 or u64), so if the ControlData or ResetTable file size claims to be over 2GB, the int will be negative and the memory requested will either be 2-4 gigabytes (32-bit size_t) or 15-16 exabytes (64-bit size_t). It's unlikely this memory request will ever be fulfilled, but if it succeeded, the subsequent read() call would fail and the memory would be deallocated and the file rejected.

However, this report reveals a more concerning issue: when the ControlData or ResetTable file length is less than 2GB, but still a large value, and there is genuinely that much data provided for the file (so read() succeeds), it allows the file creator to make libmspack allocate and use up to 4GB of memory. This is against library user expectations, so the upper limit on memory use for these system files should be quantified.

read_sys_file() is used to read 3 system files into memory: SpanInfo, ControlData and ResetTable. SpanInfo is already rejected if its file length is not exactly 8 bytes.

ControlData is not loaded if its file length is less than 28 bytes, but larger files were allowed in case a new version of the file was introduced. As no new version has been introduced in years, it is much better to reject the file if not exactly 28 bytes. This does not affect any genuine CHM files.

ResetTable can theoretically be huge; the longest possible LZX stream (16 exabytes) could have a 4 petabyte ResetTable. But practically, the largest seen in the wild is 46 kilobytes (PHP manuals). so I picked an arbitrary upper limit of 1MB, which allows for decompressed LZX streams of over 4GB.

This is fixed in commit 2c4bf97, please confirm.

from libmspack.

Thanks for your very detailed analysis!

I have tried the newest commit, this bug has been fixed. Good work!

from libmspack.