Comments (8)
I increased it to 64 MB, cos why not.
For GZIP there is original data size information that can be used to prepare buffer. Deflate streams should be streamed along with original data size.
I wouldnt worry, there is validation (CRC32 + original data size), so if you will find a valid GZ file that the zflate has problems with let me know.
from zflate.
Code to produce the error:
program project1;
{$mode objfpc}{$H+}
uses
{$IFDEF UNIX}
cmem, cthreads,
{$ENDIF}
Classes, SysUtils, zflate;
function ungzip(bytes : TBytes) : TBytes;
begin
result := zflate.zdecompress(bytes);
if zlastError <> 0 then
raise Exception.create('Failed to read compressed content: '+zflatetranslatecode(zlasterror));
end;
var
b : TBytes;
f : TFileStream;
begin
try
f := TFileStream.create('/Users/grahamegrieve/temp/package.tgz', fmOpenRead);
try
setLength(b, f.Size);
f.Read(b[0], f.size);
writeln('Unencrpyted is '+inttostr(length(ungzip(b)))+' bytes in size');
finally
f.free;
end;
except
on e : Exception do
writeln('Error: '+e.message);
end;
end.
from zflate.
reproduced on : Lazarus 3.1 (rev lazarus_3_0-15-g9bef988478) FPC 3.3.1 aarch64-darwin-cocoa and Lazarus 2.2.6 (rev 0df75f4) FPC 3.2.2 x86_64-win64-win32/win64
from zflate.
Because z.state^.mode is BAD somewhere in the middle of the file
from zflate.
Fixed by increasing buffer size (zbuffersize
) from 4 to 16 MB, if you can confirm you can close this issue
from zflate.
it does fix, thanks. But it raises multiple questions for me. Is 16 enough for everything? What is enough? Shouldn't the underlying zlib library return (say) E_OUT_OF_MEMORY rather than E_DATA_ERROR if there's not enough memory?
from zflate.
Actually I dont know why it returned Z_DATA_ERROR and not Z_BUF_ERROR, I didnt dig that deep in to Z* units.
zchunkmaxsize
and zbuffersize
are VARs and not CONSTs on purpose. To adjust them. I wanted to create a rountine that would double the buffer size in case its too small. Maybe in the future.
The case is, you can compress a very large string to a very small output, lets say you have 20 MB of "x", just "x" repeated 20 mln times. This will compress to just 20405 bytes (in case of GZIP level 9). Now, you have a buffer of 16 MB and chunk size of 128 KB, this means you get whole 20405 bytes and try to expand it to 20 MB having only 16 MB buffer, it wont work. You need to use either smaller chunk size, or bigger buffer size.
from zflate.
That's kind of unfortunate for code that's decompressing tgzs from unknown software - I don't know what buffer has to be? I mean, we can assume 16MB, I guess, and bump it up if there's ever a problem, but it kind of feels like hanging technical risk that I'd rather just not have. The paszlib code can't allocate a bigger buffer on the fly?
from zflate.
Related Issues (1)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zflate.