Can you say something about whether you release Python's GIL while the C compression/decompression/hashing code does its work? If you don't, do you think this would be possible?

It would be good for Python software that is multithreaded and wants to do stuff in parallel (like loading the next piece from disk or encrypting or ...).

See there about why I am asking:

https://github.com/thomaswaldmann/borg/tree/multithreading

I'm trying to install lz4 using pip install lz4, but I get the following error:

Collecting lz4                                                                                                 
  Using cached lz4-0.7.0.tar.gz                                                                                
Installing collected packages: lz4                                                                             
  Running setup.py install for lz4                                                                             
    Complete output from command C:\tools\python2\python.exe -c "import setuptools, tokenize;__file__='c:\\users\\xxxxx~1.xxx\\appdata\\local\\temp\\pip-build-tzepnf\\lz4\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\users\xxxxx~1.xxx\appdata\local\temp\pip-p767ra-record\install-record.txt --single-version-externally-managed --compile:                 
    running install                                                                                            
    running build                                                                                              
    running build_ext                                                                                          
    building 'lz4' extension                                                                                   
    creating build                                                                                             
    creating build\temp.win-amd64-2.7                                                                          
    creating build\temp.win-amd64-2.7\Release                                                                  
    creating build\temp.win-amd64-2.7\Release\src                                                              
    C:\Users\xxxxx.xxxxxxxx\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\tools\python2\include -IC:\tools\python2\PC /Tcsrc/lz4.c /Fobuild\temp.win-amd64-2.7\Release\src/lz4.obj -std=c99 -O3 -Wall -W -Wundef -DLZ4_VERSION="r119"                      
    cl : Command line error D8004 : '/W' requires an argument                                                  
    error: command 'C:\\Users\\xxxxx.xxxxxxxx\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\cl.exe' failed with exit status 2

I'm running Python 2.7 and have the matching Microsoft Visual C++ Compiler for Python 2.7 installed. Is this lz4 package not compatible with Python 2.7?

I have a xamarin application and its files are compressed using lz4 I could easily decompress the files it using lz4.block.decompress but I couldn't compress it again, can anyone help me with that?

I used the following script to decompress the file and I could read the content of the DLL file but when I modified something and try to recompress it again to patch the APK file I couldn't compress the file in the right format.

with open(input_filepath, "rb") as xalz_file:
    data = xalz_file
    header = data.read(8)
    if header[:4] != b"XALZ":
        sys.exit("The input file does not contain the expected magic bytes, aborting ...")

    payload = data.read()
    decompressed = lz4.block.decompress(payload)
            
    with open(output_filepath, "wb") as output_file:
        output_file.write(decompressed)
        output_file.close()
    print("result written to file")

Note that I couldn't decompress or compress the DLL file using lz4tools.

First of all, I cannot pip install it on windows, therefore I used python setup.py install. It seems that -W and -Wundef is not compatible with MSVC cl.exe so I changed to mingw32 and added a distutils.cfg at python Lib/dist
I found this still cannot be compiled and I switched to mingw32-x64
Still no luck,
Eventually I found that these two lines in setup.py
" "-DVERSION="%s"" % VERSION_STR,
"-DLZ4_VERSION="r124"",
"

needs to be:

"
"-DVERSION=\"%s\"" % VERSION_STR,
"-DLZ4_VERSION=\"r124\"",
"
Hope this helps for anyone who is having trouble installing it under windows.

When we try to decompress many files in a loop the process is killed by the OS.

Python 2.7.6
OS: Ubuntu 14.04.3 LTS 14.04

import lz4tools
file_name = "/tmp/file.lz4"
for i in range(1, 1000):
    print i
    file_ = lz4tools.open(file_name)
    data = file_.decomp
    file_.close()

Output

...
364
365
366
367
368
369
Fatal Python error: deallocating None

strace

16:14:39.809310 write(2, "Fatal Python error: deallocating"..., 38) = 38
16:14:39.810527 rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
16:14:39.814320 tgkill(69, 69, SIGABRT) = 0
16:14:39.814395 --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=69, si_uid=0} ---
16:14:39.841279 +++ killed by SIGABRT +++

/cc @mspringfeldt @bixu

Lz4File's read function not return last 1 byte

• Lz4File's self.end must be set to total (not total - 1)

I can't decompress any file and the unit test from the project is not passing.

System: OSX 10.10.5
Python 2.7.10

.Unable to load blockDict. Possibly not a lz4 file.

E

ERROR: test_2_file (main.TestLZ4File)

Traceback (most recent call last):
File "tests/test.py", line 12, in test_2_file
testTar = lz4tools.openTar('src.tar.lz4')
File "/Users/zimmermann/workspace/lz4tools/lz4tools/init.py", line 109, in openTar
return Lz4Tar.lz4open(name, 'r', fileObj)
File "/Users/zimmermann/workspace/lz4tools/lz4tools/lz4tar.py", line 20, in lz4open
lz4FileOut = Lz4File.open(fileObj=fileobj)
File "/Users/zimmermann/workspace/lz4tools/lz4tools/lz4file.py", line 39, in open
return cls(name, fileObj, seekable)
File "/Users/zimmermann/workspace/lz4tools/lz4tools/lz4file.py", line 29, in init
self.load_blocks()
File "/Users/zimmermann/workspace/lz4tools/lz4tools/lz4file.py", line 73, in load_blocks
data = self.read_block(blkSize, setCur=False)
File "/Users/zimmermann/workspace/lz4tools/lz4tools/lz4file.py", line 138, in read_block
resultDict = lz4f.decompressFrame(compData, self.dCtx, self.blkSizeID)
RuntimeError: More keyword list entries (4) than format specifiers (3)

Ran 2 tests in 0.010s

FAILED (errors=1)

Hi, could you please create tag which is defining 1.3.1.2 version what's currently in pip ? I'm building debian package and currently i'm referencing to tag v1.3 where is 1.3.1.1 version now ...
But, i need 1.3.1.2

Thanks,
Michal

When building 1.2 release with Anaconda on windows, I get the following errors

src/python-lz4f.c: In function 'initlz4f':
src/python-lz4f.c:406:51: error: too many decimal points in number
src/python-lz4f.c:407:55: error: too many decimal points in number
src/python-lz4f.c:408:55: error: 'r123' undeclared (first use in this function)
src/python-lz4f.c:408:55: note: each undeclared identifier is reported only once

Replacing the versions with their literals in the abovementioned file, like so

    PyModule_AddStringConstant(module, "VERSION", "1.2.1.2");
    PyModule_AddStringConstant(module, "__version__", "1.2.1.2");
    PyModule_AddStringConstant(module, "LZ4_VERSION", "r123");

Resolves the problem. It looks like the string literals VERSION_STR in setup.py aren't being treated as strings, but numbers instead.

With gzip.open() the following is possible:

with gzip.open(filename, 'rb') as f:
    for line in f:

How do I achieve this with lz4tools?

lz4tools.open(filename).read()

fails with RecursionError: maximum recursion depth exceeded in comparison -- I assume because it tries to load everything into memory and doesn't return a generator?

I install python 3.7 and use code below from lz4 documentation to read mozzila firefox compressed json file to decompress to json file to extract bookmarks. I get error RuntimeError: LZ4F_decompress failed with code: ERROR_frameType_unknown.

import json
import os
import lz4.frame

testfile = "C:\Users\Vartotojas\application data\mozilla\firefox\Profiles\8zco1w2p.default-1520716267057\bookmarkbackups\bookmarks-2018-06-30_23_zKtuViq4dpuCfZ3hgk47jg==.jsonlz4"
testfile1 = "testfile_1.json"

input_data = 20 * os.urandom(1024)
with lz4.frame.open('testfile1', mode='wb') as fp:
bytes_written = fp.write(input_data)
bytes_written == len(input_data)

with lz4.frame.open(testfile, mode='r') as fp:
output_data = fp.read()
output_data == input_data

output_data = fp.read()

File "C:\Users\Vartotojas\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lz4\frame_init_.py", line 635, in read
return self._buffer.read(size)
File "C:\Users\Vartotojas\AppData\Local\Programs\Python\Python37-32\lib_compression.py", line 103, in read
data = self.decompressor.decompress(rawblock, size)
File "C:\Users\Vartotojas\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lz4\frame_init.py", line 398, in decompress
return_bytearray=self._return_bytearray,
RuntimeError: LZ4F_decompress failed with code: ERROR_frameType_unknown

The standard file objects in python return an empty string when EOF is reached. See https://docs.python.org/2/tutorial/inputoutput.html#methods-of-file-objects. "If the end of the file has been reached, f.read() will return an empty string ("")."
Throwing an exception makes it so that a Lz4File can't be substituted for a regular file.