Segmentation fault when using stdin on large files/streams about lrzip HOT 8 CLOSED

By x84 machine I assume you mean x86 machine. The size you're hitting that error is almost exactly the size of a 32bit integer suggesting a 32bit overflow. Since I can't reproduce it here with a 64 bit kernel and userspace I'm going to assume your userspace +/- kernel are 32bit?

from lrzip.

By x84 machine I assume you mean x86 machine.

Sorry, I meant to type "x86_64". The machine is 64 bits and has 16 GB of RAM. Perhaps it has to do with memory size?

Same thing occurs in a docker vm running ubuntu:14.04 on the same machine. Gist of all steps @ https://gist.github.com/kylemanna/c141582cd38d83467e6d

I tried it on my laptop also running a very similarly configured Arch Linux x64 with 16 GB of RAM, it crashed at 2.6GB of input (desktop was 2.61GB). Also tried on an older x64 machine with only 4GB of ram, it crashed at 574MB of input file read.

from lrzip.

The point in to the input that it crashes at is very repeatable (rough math):

desktop: 2610 / 16036 = 0.16275879271
laptop 2600 / 15976 = 0.16274411617
htpc: 575 / 3452 = 0.16628041714

I also tried adding -w 1 to the lrzip command line, no change. I assume it has consumed the satisfactory compression window input and begins processing it where something goes wrong.

from lrzip.

Also tested on Ubuntu 12.04.5 LTS with package lrzip-0.608-1 running under VirtualBox with 8 GB of RAM and x86_64.

$ time lrzip -d  < 5GB.raw.lrz > /dev/null
Decompressing...
[2]    27365 segmentation fault (core dumped)  lrzip -d < 5GB.raw.lrz > /dev/null
lrzip -d < 5GB.raw.lrz > /dev/null  11.98s user 7.16s system 20% cpu 1:31.86 total

from lrzip.

The control->tmp_inbuf buffer that read_fdin() is attempting to write to is the size of control->in_maxlen.

When reading the lrzip header, fill_buffer() attempts to read to the end of the stream, and in the case of a large incompressible stream (see my ghetto openssl example stdin), the stream extends beyond the size of the tmp_inbuf, overflows, and crashes. See original stack trace above.

This could be alleviated by dumping the stream to the temporary file (maybe via read_1g()?), but it's not immediately obvious to me how to implement this.

from lrzip.

Finally got a chance to look at this and yes you're right. I will try to work around the way you suggested thanks.

from lrzip.

Fixed in git.

from lrzip.

Not sure if this is fixed. Just ran it and run in to two problems:

1. With a 5GB file it runs the system out of memory when $TMPDIR points to a tmpfs. Perhaps it should check if it's about to write to a tmpfs and fail if it's true. This sounds like another issue.
2. With a 3GB file it failed to compare the md5sum embedded in the lrzip metadata. Running lrzip on the file directly (i.e. no streams) it works as expected.

from lrzip.