Comments (8)
By x84 machine I assume you mean x86 machine. The size you're hitting that error is almost exactly the size of a 32bit integer suggesting a 32bit overflow. Since I can't reproduce it here with a 64 bit kernel and userspace I'm going to assume your userspace +/- kernel are 32bit?
from lrzip.
By x84 machine I assume you mean x86 machine.
Sorry, I meant to type "x86_64". The machine is 64 bits and has 16 GB of RAM. Perhaps it has to do with memory size?
Same thing occurs in a docker vm running ubuntu:14.04 on the same machine. Gist of all steps @ https://gist.github.com/kylemanna/c141582cd38d83467e6d
I tried it on my laptop also running a very similarly configured Arch Linux x64 with 16 GB of RAM, it crashed at 2.6GB of input (desktop was 2.61GB). Also tried on an older x64 machine with only 4GB of ram, it crashed at 574MB of input file read.
from lrzip.
The point in to the input that it crashes at is very repeatable (rough math):
desktop: 2610 / 16036 = 0.16275879271
laptop 2600 / 15976 = 0.16274411617
htpc: 575 / 3452 = 0.16628041714
I also tried adding -w 1
to the lrzip command line, no change. I assume it has consumed the satisfactory compression window input and begins processing it where something goes wrong.
from lrzip.
Also tested on Ubuntu 12.04.5 LTS with package lrzip-0.608-1 running under VirtualBox with 8 GB of RAM and x86_64.
$ time lrzip -d < 5GB.raw.lrz > /dev/null
Decompressing...
[2] 27365 segmentation fault (core dumped) lrzip -d < 5GB.raw.lrz > /dev/null
lrzip -d < 5GB.raw.lrz > /dev/null 11.98s user 7.16s system 20% cpu 1:31.86 total
from lrzip.
The control->tmp_inbuf
buffer that read_fdin()
is attempting to write to is the size of control->in_maxlen.
When reading the lrzip header, fill_buffer()
attempts to read to the end of the stream, and in the case of a large incompressible stream (see my ghetto openssl example stdin), the stream extends beyond the size of the tmp_inbuf, overflows, and crashes. See original stack trace above.
This could be alleviated by dumping the stream to the temporary file (maybe via read_1g()
?), but it's not immediately obvious to me how to implement this.
from lrzip.
Finally got a chance to look at this and yes you're right. I will try to work around the way you suggested thanks.
from lrzip.
Fixed in git.
from lrzip.
Not sure if this is fixed. Just ran it and run in to two problems:
- With a 5GB file it runs the system out of memory when
$TMPDIR
points to atmpfs
. Perhaps it should check if it's about to write to a tmpfs and fail if it's true. This sounds like another issue. - With a 3GB file it failed to compare the md5sum embedded in the lrzip metadata. Running lrzip on the file directly (i.e. no streams) it works as expected.
from lrzip.
Related Issues (20)
- Pretty Format lrzip -i HOT 1
- Could -U use an SSD as if it was a RAM drive? HOT 3
- lrzip -L9 performs worse than lrzip -L8 HOT 3
- Deallocation of control->suffix corrupts Heap Memory HOT 5
- #216 may have other issues HOT 1
- Code Cleanup: remove function rzip.c: rzip_control_free() HOT 1
- Name clash with lrzsz Zmodem/Xmodem package HOT 2
- util.h: failure and fatal macros always fatal_exit
- Error when trying to compress, please help HOT 2
- UndefinedBehaviorSanitizer: invalid shifts
- Remove doxygen doc build and Lrzip.h
- Incorrect number of arguments passed to print_maxverbose in lz4_compresses
- How to read the output HOT 1
- lrzip -t file.lrz fails when run from write-protected dir
- Streaming issue "No space left on device" in lrzip 0.651 HOT 1
- "Warning, low memory for chosen compression settings" for small target files HOT 12
- lrzip 0.651 test issue HOT 3
- Use of uninitialized memory bug HOT 1
- 41e8014 Add a -Q/--very-quiet option: Suppresses INFO. It shouldn't
- memory error in fill_buffer lrzip/stream.c HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lrzip.