Comments (15)
Tracked it down to a failing zlib function deflateInit2_
. It frequently returns Z_STREAM_ERROR
This condition causes decompression error.
if (memLevel < 1 || memLevel > MAX_MEM_LEVEL || method != Z_DEFLATED ||
windowBits < 8 || windowBits > 15 || level < 0 || level > 9 ||
strategy < 0 || strategy > Z_FIXED || (windowBits == 8 && wrap != 1)) {
return Z_STREAM_ERROR;
}
Namely windowBits == 8 && wrap != 1
fails. Values are usually around that check are usually
windowBits = 8; wrap = 0
windowBits = 8; wrap = 2
but the variables are tweaked before the check. Input value into the function is windowBits = 24
.
This additional check was intruduced in madler/zlib@049578f
from zlib.
Skipping 8-bit window seems to make tests pass. A crude hack I used locally:
diff --git a/test/Test.hs b/test/Test.hs
index bbd8b94..81bd260 100644
--- a/test/Test.hs
+++ b/test/Test.hs
@@ -66,2 +66,3 @@ prop_decompress_after_compress w cp dp =
-- of the zlib source code.
+ compressWindowBits cp /= WindowBits 8 &&
(decompressWindowBits dp > compressWindowBits cp || decompressWindowBits dp > WindowBits 8) &&
@@ -76,2 +77,3 @@ prop_gziporzlib1 cp dp =
decompressWindowBits dp > compressWindowBits cp &&
+ compressWindowBits cp /= WindowBits 8 &&
decompressBufferSize dp > 0 && compressBufferSize cp > 0 ==>
@@ -85,2 +87,3 @@ prop_gziporzlib2 cp dp =
decompressWindowBits dp >= compressWindowBits cp &&
+ compressWindowBits cp /= WindowBits 8 &&
decompressBufferSize dp > 0 && compressBufferSize cp > 0 ==>
@@ -94,2 +97,3 @@ prop_gzip_concat cp dp input =
decompressWindowBits dp >= compressWindowBits cp &&
+ compressWindowBits cp /= WindowBits 8 &&
decompressBufferSize dp > 0 && compressBufferSize cp > 0 ==>
from zlib.
@trofi was there any significant (API?) change in the zlib 1.2.11 which could explain this?
PS: I just tried with zlib-1.2.9
which appears to be the first release that broke the decompress . compress
tests. And unfortunately, zlib-1.2.9 contains over 3.5 years worth of changes:
Changes in 1.2.9 (31 Dec 2016)
- Fix contrib/minizip to permit unzipping with desktop API [Zouzou]
- Improve contrib/blast to return unused bytes
- Assure that gzoffset() is correct when appending
- Improve compress() and uncompress() to support large lengths
- Fix bug in test/example.c where error code not saved
- Remedy Coverity warning [Randers-Pehrson]
- Improve speed of gzprintf() in transparent mode
- Fix inflateInit2() bug when windowBits is 16 or 32
- Change DEBUG macro to ZLIB_DEBUG
- Avoid uninitialized access by gzclose_w()
- Allow building zlib outside of the source directory
- Fix bug that accepted invalid zlib header when windowBits is zero
- Fix gzseek() problem on MinGW due to buggy _lseeki64 there
- Loop on write() calls in gzwrite.c in case of non-blocking I/O
- Add --warn (-w) option to ./configure for more compiler warnings
- Reject a window size of 256 bytes if not using the zlib wrapper
- Fix bug when level 0 used with Z_HUFFMAN or Z_RLE
- Add --debug (-d) option to ./configure to define ZLIB_DEBUG
- Fix bugs in creating a very large gzip header
- Add uncompress2() function, which returns the input size used
- Assure that deflateParams() will not switch functions mid-block
- Dramatically speed up deflation for level 0 (storing)
- Add gzfread(), duplicating the interface of fread()
- Add gzfwrite(), duplicating the interface of fwrite()
- Add deflateGetDictionary() function
- Use snprintf() for later versions of Microsoft C
- Fix *Init macros to use z_ prefix when requested
- Replace as400 with os400 for OS/400 support [Monnerat]
- Add crc32_z() and adler32_z() functions with size_t lengths
- Update Visual Studio project files [AraHaan]
Changes in 1.2.8 (28 Apr 2013)
from zlib.
SO basically the problems seems to be that compressing the empty string ""
fails now?
from zlib.
There was no existing API change (a few new functions were added).
But stream validation was updated. Like here:
madler/zlib@b516b4b
from zlib.
Looks like "" is a miniman shrunk case. If we filter null out tests still fail:
prop_decompress_after_compress :: Format
-> CompressParams
-> DecompressParams
+ -> BL.ByteString
-> Property
-prop_decompress_after_compress w cp dp =
+prop_decompress_after_compress w cp dp i =
(w /= zlibFormat || decompressWindowBits dp >= compressWindowBits cp) &&
-- Zlib decompression has been observed to fail with both compress and decompress
-- window bits = 8. This seems to be contrary to the docs and to a quick reading
-- of the zlib source code.
(decompressWindowBits dp > compressWindowBits cp || decompressWindowBits dp > WindowBits 8) &&
- decompressBufferSize dp > 0 && compressBufferSize cp > 0 ==>
- liftM2 (==) (decompress w dp . compress w cp) id
+ decompressBufferSize dp > 0 && compressBufferSize cp > 0 && (not $ BL.null i) ==>
+ (decompress w dp $ compress w cp $ i) == i
zlib tests
property tests
decompress . compress = id (standard): FAIL (0.42s)
*** Failed! Exception: 'user error (Codec.Compression.Zlib: stream error)' (after 31 tests and 10 shrinks):
GZip
CompressParams {compressLevel = CompressionLevel 8, compressMethod = Deflated, compressWindowBits = WindowBits 8, compressMemoryLevel = MemoryLevel 8, compressStrategy = DefaultStrategy, compressBufferSize = 124143, compressDictionary = Nothing}
DecompressParams {decompressWindowBits = WindowBits 15, decompressBufferSize = 293676, decompressDictionary = Nothing, decompressAllMembers = False}
"\NUL"
from zlib.
I wonder if the Compress/Decompress parameters must match more closely w/ zlib-1.2.11 than they did before. specifically the window-bits
from zlib.
@trofi I tried reproducing the issue in GHCi w/ zlib-1.2.11 using the code from the quickcheck test... but I can't seem to reproduce this. Is this even deterministic?
from zlib.
We're not the only ones having problems w/ zlib-1.2.{9,10,11}:
from zlib.
Hmm, nasty.
from zlib.
So the ocaml people tracked their bug down to the fact that they were keeping the z_stream
C struct in movable memory on the ocaml heap, and a new pointer sanity check in the zlib C code was failing. xavierleroy/camlzip#1 (comment)
However we already use a ForeignPtr
and mallocForeignPtrBytes
for the z_stream
so relocation would not appear to be the problem we're having. Someone may need to track down exactly which new check in the C code is being triggered.
from zlib.
@trofi good work tracking it down! Would you like to make a PR for upstream?
from zlib.
@trofi actually having read the zlib change that you pointed me to, I've decided to do it slightly differently: just change the range of valid window bits from 8..15 to 9..15. It seems 8 just isn't supported at all: it fails for raw and gzip and is translated to 9 for zlib format.
from zlib.
@dcoutts @trofi regarding your observation that ws=8 is buggy, see also madler/zlib#94
from zlib.
@dcoutts - it seems that this ticket is now fixed? Given you changed the window bits range?
from zlib.
Related Issues (20)
- c code bundled in zlib is vulnerable HOT 1
- GHC 8.10.1 support HOT 1
- Upstream error when building with GHC 8.10.2, on Windows 10 HOT 1
- stack fails to build this as a dependency for at least all stackage LTS versions greater than or equals 15.1 HOT 3
- Cannot find foreign library (Ubuntu, with zlib1g-dev installed) HOT 2
- ghc.exe: could not execute: C:/GitLabRunner/builds/2WeHDSFP/0/ghc/ghc/inplace/mingw/bin/ld.exe HOT 1
- Support for bytestring 0.11 HOT 4
- Increase Cabal upper bound to <3.3 HOT 1
- Release version of zlib for GHC 9.0
- [s390x] testsuite fails on Fedora 36 HOT 3
- Support bytestring chunks bigger than 4Gb HOT 2
- Build failure on GHC 8.x (macOS 10.14): ld: unknown option: -no_fixup_chains HOT 6
- unknown symbol `inflateReset' HOT 10
- RLE is useful HOT 1
- tasty-1.5 not accepted HOT 1
- `hsc2hs` not declared as build tool dependency HOT 2
- Drop GHC 6 ? HOT 1
- Enable non-blocking-ffi by default, provide blocking FFI as an option HOT 1
- Potential issue building on Windows HOT 3
- Runtime failure on Windows on version 0.7.0.0 HOT 29
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zlib.