kaitz / paq8pxd Goto Github PK

Use struct's for all detectable data types. Examples are TextInfo and bmpInfo.

https://encode.su/threads/1464-Paq8pxd-dict?p=66290&viewfull=1#post66290

paq8pxd has a small bug in the detection routine: it may happen when it skips an image -> it will stop detecting more.
https://encode.su/threads/1464-Paq8pxd-dict?p=70995&viewfull=1#post70995

I really like this compressor and was just wondering if you planned on updating it still.

There are some variables common. So when two threads try to use wrt transform then program will crash with SIGSEV.

So something I've noticed during my tests is that the overall memory allocated to paq8pxd does not match up with what is being stated at the end when it finished.

Two examples compressed individually: one JPEG and one uncompressed 24-bit image.

Baseline JPEG (-s9):
Task Manager and Process Explorer reports 2.8GiB allocated.
Reported memory usage by paq8pxd is: 3712 MB (3893306956 bytes)

Uncompressed 24-bit Image (-s9):
Task Manager and Process Explorer reports 1.2GiB allocated.
Reported memory usage by paq8pxd is: 3764 MB (3947871882 bytes)

It gets even wilder at (-s15) where I can manage to trigger and "Out of Memory" error at the very end of the compression if I start doing other things on my system while it's running. This is why I'm thinking it's some kind of memory allocation issue.

After hearing that it would be difficult for files above 2GiB to be supported, I'd figure I'd leave this here for reference for anyone to comment on.

paq8pxd, and by extension the other paq8 variants, have an incredibly powerful preprocessor that would benefit from having large file & archive support above 2GiB. This way you can use paq8pxd to preprocess your data and use a different compression scheme than the one included.

I'll be realistic here and say that I do not expect this to happen.

P.S. I am not a programmer so please don't berate me with comments like "add it yourself."

computer@computer-Mac-mini paq8pxd-master % g++ paq8pxd.cpp -O2 -DUNIX -s -lz -o paq8pxd
paq8pxd.cpp:694:50: warning: format specifies type 'unsigned long' but the argument has type 'unsigned long long' [-Wformat]
double(clock()-start_time)/CLOCKS_PER_SEC, ((maxmem)/1024)/1024,(maxmem));
^~~~~~~~~~~~~~~~~~~~
paq8pxd.cpp:694:71: warning: format specifies type 'unsigned long' but the argument has type 'U64' (aka 'unsigned long long') [-Wformat]
double(clock()-start_time)/CLOCKS_PER_SEC, ((maxmem)/1024)/1024,(maxmem));
^~~~~~~~
paq8pxd.cpp:2076:7: warning: expression result unused [-Wunused-value]
((U32 *) &tx[nx],a);
^ ~~~~~~~
paq8pxd.cpp:2076:23: warning: expression result unused [-Wunused-value]
((U32 *) &tx[nx],a);
^
paq8pxd.cpp:2081:7: warning: expression result unused [-Wunused-value]
((U64 *) &tx[nx],a);
^ ~~~~~~~
paq8pxd.cpp:2081:23: warning: expression result unused [-Wunused-value]
((U64 *) &tx[nx],a);
^
In file included from paq8pxd.cpp:2786:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/immintrin.h:40:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/wmmintrin.h:15:
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/__wmmintrin_aes.h:36:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/__wmmintrin_aes.h:56:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/__wmmintrin_aes.h:76:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/__wmmintrin_aes.h:96:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/__wmmintrin_aes.h:113:1: error: function definition is not allowed here
{
^
In file included from paq8pxd.cpp:2786:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/immintrin.h:44:
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/clflushoptintrin.h:21:34: error: function definition is not allowed here
_mm_clflushopt(void const * __m) {
^
In file included from paq8pxd.cpp:2786:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/immintrin.h:48:
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/clwbintrin.h:32:27: error: function definition is not allowed here
_mm_clwb(void const *__p) {
^
In file included from paq8pxd.cpp:2786:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/immintrin.h:52:
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:61:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:79:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:97:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:115:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:134:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:153:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:171:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:189:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:208:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:227:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:246:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:265:1: error: function definition is not allowed here
{
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
6 warnings and 20 errors generated.

Right now they can only be found on the forum. It would be great if you could add them to the releases here.

A way to check archives without extracting them or somewhere along those lines.

The only way to verify the validity of an archive right now is to extract it and compare it to the original manually.

Detecting MSZIP streams is present.
https://stackoverflow.com/questions/39390314/deflating-data-from-mszip-format
https://blogs.kde.org/2008/01/04/kcabinet-mostly-working

I'm attempting to preprocess (-s0) compact disc images stored in (BIN+CUE) or (BIN+GDI) formats and paq8pxd v104 is producing a corrupted output.

paq8px v205fix1 doesn't seem to have this problem so I've been using that for now, but I'm running into the file size limit of 2GiB which is too limiting for what I want to use it for.

I am running Windows 10 and using the provided binary here on the github.

As a side note, anything I extract always produces this message:
"DeCompressing FileDisk: unable to open file (No such file or directory)"

LZWDecode

$ g++ paq8pxd.cpp -DWINDOWS -DMT -msse2 -O3 -s -static -o paq8pxd.exe
paq8pxd.cpp:554:0: warning: "MT" redefined
#define MT //uncomment for multithreading, compression only

:0:0: note: this is the location of the previous definition
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17588): undefined reference to inflateInit_' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17630): undefined reference to deflateInit2_'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x1787a): undefined reference to deflate' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x178f0): undefined reference to inflate'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17bc5): undefined reference to deflateEnd' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17be0): undefined reference to inflateEnd'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17de5): undefined reference to inflateInit_' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17e8c): undefined reference to inflate'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17f6c): undefined reference to deflateEnd' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17fc7): undefined reference to inflateInit2_'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x18022): undefined reference to inflateInit2_' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x182c2): undefined reference to deflateInit2_'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x183ad): undefined reference to deflate' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x184a4): undefined reference to deflateEnd'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x2845a): undefined reference to inflate' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x28469): undefined reference to inflateEnd'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x2b332): undefined reference to inflate' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x2b376): undefined reference to inflateEnd'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0xc773): undefined reference to inflateInit_' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0xc793): undefined reference to inflateInit2_'
collect2.exe: error: ld returned 1 exit status

I found a file (possibly damaged) that doesn't restore losslessly

ati3duag.dl_.zip

EDIT: This is NOT the correct file; see the update.

Hi, I'm testing this utility on a Windows 11 ARM VM and I can't get it to work, I tried to drag and drop a folder with an handful of jpegs inside, but the terminal window briefly appears and then automatically closes with no result.

If I manually call the script and pass the arguments it just exists with no output:

paq8pxd_v106.exe C:\test-folder test.paq8pxd

paq8pxd_v106.exe .\test-folder .\test.paq8pxd

paq8pxd_v106.exe C:\test-folder C:\test.paq8pxd

paq8pxd_v106.exe C:\test-folder

None of the above produced any output. What am I doing wrong?

I also tried with v105 but I get the same result.

Hi! This is an absolutely amazing compression program. It is the best ratio I've ever seen.

You say that files over 2 GB are not supported.

Do you think a 9 GB file would be possible to compress? Or would I be better off breaking it up using 7zip and compressing the pieces?

Why? It makes it harder to find and/or fix issues if all the code is in a one file. Is it for perfomance (somehow)? Was the one-file style borrowed from older codebase (AFAIK paq1 is 22 years old)?