dcwatson / deflate Goto Github PK
View Code? Open in Web Editor NEWPython extension wrapper for libdeflate.
License: MIT License
Python extension wrapper for libdeflate.
License: MIT License
https://github.com/dcwatson/deflate/runs/5433169119?check_suite_focus=true
this was running for the 0.3.0 tag after I tagged it.
do we need a timeout there?
I first noticed this in CI:
____________________________ test_file_header[True] ____________________________
use_deflate = True
@pytest.mark.parametrize("use_deflate", [False, True])
def test_file_header(use_deflate):
if use_deflate:
> pytest.importorskip("deflate")
E pytest.PytestDeprecationWarning:
E Module 'deflate' was found, but when imported by pytest it raised:
E ImportError("dlopen(/Users/runner/miniconda3/envs/test/lib/python3.9/site-packages/deflate.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_libdeflate_arm_cpu_features'")
E In pytest 9.1 this warning will become an error by default.
E You can fix the underlying problem, or alternatively overwrite this behavior and silence this warning by passing exc_type=ImportError explicitly.
E See https://docs.pytest.org/en/stable/deprecations.html#pytest-importorskip-default-behavior-regarding-importerror
(perhaps because GitHub just upgraded to ARM/M1/M2 Macs), but I also see it on a (M2) Mac laptop:
>>> import deflate
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/jpivarski/mambaforge/lib/python3.10/site-packages/deflate.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_libdeflate_arm_cpu_features'
This is deflate 0.6.0, freshly installed from PyPI.
My version of libdeflate is 1.17, installed through conda. (Overall, it's a conda environment, with deflate being the only package installed by pip within that environment.)
Thanks for writing this wrapper, is it possible to use it to stream decompress chunked data? Either through a pipe or streaming HTTP response?
Thanks again.
Add libdeflate gunzip logic to handle gzipped files which decompress to files larger than 4G or which contain multiple concatenated gzip files.
Some gzip files have 0 for the decompressed gzip length or contain multiple gzip files concatenated together so the size hint is only for the last gzip "file" instead of for the whole file.
https://github.com/ebiggers/libdeflate/blob/master/programs/gzip.c#L206-L273
Please stay up to date with the latest version of libdeflate so that people don't run into issues that have already been fixed. Thanks!
after adding testing to the automated wheel building running via github actions, i noticed that the pypy wheel does not test ok. likely no new issue, just nobody noticed yet / complained yet.
when this issue is fixed, the pypy wheel building can be re-enabled.
see there:
Hi, I work on python-isal, which wraps ISA-L. It also aims to accelerate compression/decompression and it supports streaming features.
Unfortunately ISA-L only works well on x86-64 (Intel, AMD) so it is much more limited than deflate in that respect.
Given that you probably work on this library because of some compression/decompression needs, I wanted to let you know about python-isal. Also I wanted to say hi, as another coder working on python bindings for a deflate-compatible compression library.
we could easily run the tests (see #15) via github actions, so that they are automatically executed for any commit / pull request.
would you like a PR for that?
I managed to build deflate under Windows with MSVC. I'm not an Python extensions expert so I'll just leave some pointers in case you decide to add official Windows support.
I had to make the following changes:
setup.py:
result = subprocess.run(["nmake", "/f", "Makefile.msc", "/a"], cwd=libdeflate_dir)
...
"deflate", sources=["deflate.c"], extra_objects=["libdeflate/libdeflatestatic.lib"]
Furthermore, MSVC's compiler breaks on:
deflate.c(72): error C2036: 'void *': unknown size
I'm not sure what the correct fix is, but I changed L72 to
bytes = (uint8_t *)(data.buf) + (data.len - 4);
to get it to compile.
Hi! I think I found the existence of a memory leak somewhere in the library, but I haven't established where yet. I only know that memory snapshots indicated that deflate.gzip_decompress
was being called and the memory was growing, memory profile plots were spiking, and when I uninstalled deflate and used gzip the problem disappeared.
I tried to reproduce the issue in a simpler environment with deflate isolated and was not successful.
Deflate is called here: https://github.com/seung-lab/cloud-files/blob/master/cloudfiles/compression.py#L184
Memory snapshots:
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=66.4 KiB, count=939, average=72 B
/home/miniconda3/envs/ig/lib/python3.8/threading.py:244: size=63.8 KiB, count=217, average=301 B
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/_helpers.py:229: size=54.0 KiB, count=108, average=512 B
/home/miniconda3/envs/ig/lib/python3.8/email/message.py:484: size=41.0 KiB, count=674, average=62 B
Extracting Locations: 0%|▍ | 500/143468 [00:04<19:15, 123.69it/s]/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=83.5 MiB, count=501, average=171 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=21.3 MiB, count=1000, average=21.8 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/_helpers.py:229: size=92.5 KiB, count=185, average=512 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=86.2 KiB, count=1248, average=71 B
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=81.3 KiB, count=685, average=122 B
Extracting Locations: 1%|▊ | 1000/143468 [00:10<23:06, 102.78it/s]/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=225 MiB, count=501, average=459 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=80.0 MiB, count=1500, average=54.6 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/_helpers.py:229: size=80.5 KiB, count=161, average=512 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=79.8 KiB, count=1147, average=71 B
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=78.0 KiB, count=661, average=121 B
Extracting Locations: 1%|█▏ | 1500/143468 [00:21<30:54, 76.56it/s]/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=199 MiB, count=501, average=408 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=132 MiB, count=2000, average=67.7 KiB
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=73.2 KiB, count=626, average=120 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=70.9 KiB, count=1008, average=72 B
/home/miniconda3/envs/ig/lib/python3.8/tracemalloc.py:532: size=68.5 KiB, count=1342, average=52 B
Extracting Locations: 1%|█▋ | 2000/143468 [00:30<35:20, 66.73it/s]/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=203 MiB, count=501, average=415 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=185 MiB, count=2500, average=75.9 KiB
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=75.8 KiB, count=644, average=120 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=75.2 KiB, count=1077, average=72 B
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/_helpers.py:229: size=72.0 KiB, count=144, average=512 B
Extracting Locations: 2%|██ | 2500/143468 [00:40<38:12, 61.50it/s]/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=241 MiB, count=3000, average=82.2 KiB
/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=214 MiB, count=501, average=437 KiB
/mnt/data02/ws9/cloud-volume/cloudvolume/datasource/precomputed/spatial_index.py:166: size=883 KiB, count=5674, average=159 B
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=75.6 KiB, count=644, average=120 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=75.3 KiB, count=1078, average=72 B
Extracting Locations: 2%|██▍ | 3000/143468 [00:50<40:38, 57.60it/s]/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=292 MiB, count=3500, average=85.5 KiB
/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=198 MiB, count=501, average=405 KiB
/mnt/data02/ws9/cloud-volume/cloudvolume/datasource/precomputed/spatial_index.py:166: size=927 KiB, count=6432, average=148 B
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=70.3 KiB, count=604, average=119 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=65.2 KiB, count=920, average=73 B
Extracting Locations: 2%|██▊ | 3500/143468 [01:00<41:49, 55.78it/s]/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=345 MiB, count=4000, average=88.4 KiB
/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=210 MiB, count=501, average=428 KiB
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=75.3 KiB, count=632, average=122 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=72.1 KiB, count=1027, average=72 B
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/_helpers.py:229: size=66.0 KiB, count=132, average=512 B
With deflate installed (black) vs uninstalled (red,blue):
I'll take a look through the code and see if I can find anything. One thing to keep in mind is that it's being executed in a thread: https://github.com/seung-lab/cloud-files/blob/master/cloudfiles/cloudfiles.py#L177
guess we want to adjust so both are the same.
Hi!
Happy holidays!
I've been just doing a routine update of all my libraries and dependency audit for Python 3.12. Would it be possible to get Python 3.12 wheels uploaded?
Thank you so much for all the work you've put into this!
Will
Hi Dan,
greetings from borgbackup development!
We currently discuss about getting rid of some code we have bundled and currently there is only crc32 code left.
We were searching where we could get some fast crc32 from and found libdeflate
and your python wrapper for it.
Discussion: borgbackup/borg#6316
Do you think a crc32 api could be added, so we could easily use it?
I've got a segmentation fault using the function deflate_decompress. Here is the test used:
(.env) @:~/wkpy/test$ python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import base64
>>> import deflate
>>> tt1 = "Sy1LzNFQt7dT10uvKs1Lzs8tKEotLtZIr8rMS8tJLEnVSEosTjUziU9JT\x635PSdUoLikqSi3TU\x43kuKTHQ\x42\x41Fr\x41\x41\x3d\x3d"
>>> tt1_decoded = base64.b64decode( tt1 )
>>> deflate.gzip_decompress( tt1_decoded )
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
deflate.DeflateError: Invalid gzip data.
>>> deflate.deflate_decompress( tt1_decoded )
Segmentation fault (core dumped)
I was trying to decompress a file found in a php file, apparently compressed with the gzdeflate() function.
it would be nice if the changesets corresponding to a pypi release would be tagged like e.g. 0.3.0
.
would be cool to have some tests for the code, like:
decompress(compress(data)) == data
for misc. data values)pytest
is a nice testing frameworktodo?:
gzip
/ zlib
python stdlib code)pytest-benchmark
is nice for benchmarks.Can't be done / out of scope:
deflate_compress(data) == gzip_compress(data)
for misc. data values)Thank you for this wonderful library! While I was experimenting with it in miniconda with python3.8 I ran into an error.
The line that fails appears to be part of Anaconda and not part of the usual compilation chain and relates to libdeflate.a being a static library.
The problem seems to exist on Ubuntu 16.04. It compiles fine on my macbook.
gcc -pthread -shared -B $MY_DIRECTORY/compiler_compat -$LMY_DIRECTORY/lib -Wl,-rpath=$MY_DIRECTORY/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/deflate.o libdeflate/libdeflate.a -o /mnt/data02/ws9/deflate/deflate.cpython-38-x86_64-linux-gnu.so
Full trace:
ERROR: Command errored out with exit status 1:
command: $MY_DIRECTORY/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/mnt/data02/ws9/deflate/setup.py'"'"'; __file__='"'"'/mnt/data02/ws9/deflate/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
cwd: /mnt/data02/ws9/deflate/
Complete output (51 lines):
running develop
running egg_info
writing deflate.egg-info/PKG-INFO
writing dependency_links to deflate.egg-info/dependency_links.txt
writing top-level names to deflate.egg-info/top_level.txt
reading manifest file 'deflate.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'deflate.egg-info/SOURCES.txt'
running build_ext
CC lib/deflate_decompress.o
CC lib/utils.o
CC lib/arm/cpu_features.o
CC lib/x86/cpu_features.o
CC lib/deflate_compress.o
CC lib/adler32.o
CC lib/zlib_decompress.o
CC lib/zlib_compress.o
CC lib/crc32.o
CC lib/gzip_decompress.o
CC lib/gzip_compress.o
AR libdeflate.a
CC lib/deflate_decompress.shlib.o
CC lib/utils.shlib.o
CC lib/arm/cpu_features.shlib.o
CC lib/x86/cpu_features.shlib.o
CC lib/deflate_compress.shlib.o
CC lib/adler32.shlib.o
CC lib/zlib_decompress.shlib.o
CC lib/zlib_compress.shlib.o
CC lib/crc32.shlib.o
CC lib/gzip_decompress.shlib.o
CC lib/gzip_compress.shlib.o
CCLD libdeflate.so.0
LN libdeflate.so
GEN programs/config.h
CC programs/gzip.o
CC programs/prog_util.o
CC programs/tgetopt.o
CCLD gzip
LN gunzip
building 'deflate' extension
creating build
creating build/temp.linux-x86_64-3.8
gcc -pthread -B $MY_DIRECTORY/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -$IMY_DIRECTORY/include/python3.8 -c deflate.c -o build/temp.linux-x86_64-3.8/deflate.o
gcc -pthread -shared -B $MY_DIRECTORY/compiler_compat -$LMY_DIRECTORY/lib -Wl,-rpath=$MY_DIRECTORY/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/deflate.o libdeflate/libdeflate.a -o /mnt/data02/ws9/deflate/deflate.cpython-38-x86_64-linux-gnu.so
$MY_DIRECTORY/compiler_compat/ld: libdeflate/libdeflate.a(deflate_decompress.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
$MY_DIRECTORY/compiler_compat/ld: libdeflate/libdeflate.a(deflate_compress.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
$MY_DIRECTORY/compiler_compat/ld: libdeflate/libdeflate.a(crc32.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
$MY_DIRECTORY/compiler_compat/ld: final link failed: nonrepresentable section on output
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
----------------------------------------
i've seen there is a newer release, maybe we should use it here?
libdeflate
exposes both libdeflate_gzip_compress
and libdeflate_zlib_compress
, but you only currently wrap the gzip one. Any plans to also wrap the zlib one?
Is this as simple as replacing libdeflate_gzip_compress with libdeflate_zlib_compress? Would seem straightforward (if verbose) to copy the bindings again for zlib?
Hi again! Sorry to bug you. I was compiling a module against Windows 64-bit with Python 3.6, 3.7, and 3.8 on AppVeyor that includes deflate. Deflate appeared to successfully install, but was not found as a module. Here's my output:
uild started
git clone -q --branch=main https://github.com/seung-lab/mapbuffer.git C:\projects\mapbuffer
git checkout -qf 8d71424c63cb13c9babbdf307ff2d2ca6e1e4d31
Running Install scripts
%PYTHON%\python.exe -m pip install wheel
Collecting wheel
Downloading https://files.pythonhosted.org/packages/c9/0b/e0fd299d93cd9331657f415085a4956422959897b333e3791dde40bd711d/wheel-0.36.1-py2.py3-none-any.whl
Installing collected packages: wheel
WARNING: The script wheel.exe is installed in 'C:\Python38-x64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed wheel-0.36.1
WARNING: You are using pip version 19.3.1; however, version 20.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
%PYTHON%\python.exe -m pip install -r requirements.txt -r requirements_dev.txt
Collecting brotli
Downloading https://files.pythonhosted.org/packages/63/29/1b104b5915e61d9f7443889657d93937b7e0b33b331609b82693547934a0/Brotli-1.0.9-cp38-cp38-win_amd64.whl (365kB)
Collecting deflate>=0.2.0
Downloading https://files.pythonhosted.org/packages/c2/42/49d6f7eecca4768e045b5bf25c8b7f650ffd88b5c0d927c5833161c496c2/deflate-0.2.0-cp38-cp38-win_amd64.whl
Collecting numpy
Downloading https://files.pythonhosted.org/packages/40/db/5060f18b0116f00ee73f8365efc9c95bd5496946290b0e7c97b6ee89dffe/numpy-1.19.4-cp38-cp38-win_amd64.whl (13.0MB)
Collecting tqdm
Downloading https://files.pythonhosted.org/packages/8a/54/115f0c28a61d56674c3a5e05c46d6c3523ad196e1dcd3e2d8b119026df36/tqdm-4.54.1-py2.py3-none-any.whl (69kB)
Collecting zstandard
Downloading https://files.pythonhosted.org/packages/1d/60/888086d0239520c087da33ca2a2f1c185b24ce1d882ba68abbd8787c7884/zstandard-0.14.1-cp38-cp38-win_amd64.whl (551kB)
Collecting pytest
Downloading https://files.pythonhosted.org/packages/b1/ee/53945d50284906adb1e613fabf2e1b8b25926e8676854bb25b93564c0ce7/pytest-6.1.2-py3-none-any.whl (272kB)
Collecting cffi>=1.11
Downloading https://files.pythonhosted.org/packages/f6/d4/bf5516922cc914174a936e7094d5f136ee4178b3509069dd3da8e3040a1a/cffi-1.14.4-cp38-cp38-win_amd64.whl (179kB)
Collecting attrs>=17.4.0
Downloading https://files.pythonhosted.org/packages/c3/aa/cb45262569fcc047bf070b5de61813724d6726db83259222cd7b4c79821a/attrs-20.3.0-py2.py3-none-any.whl (49kB)
Collecting iniconfig
Downloading https://files.pythonhosted.org/packages/9b/dd/b3c12c6d707058fa947864b67f0c4e0c39ef8610988d7baea9578f3c48f3/iniconfig-1.1.1-py2.py3-none-any.whl
Collecting toml
Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl
Collecting packaging
Downloading https://files.pythonhosted.org/packages/28/87/8edcf555adaf60d053ead881bc056079e29319b643ca710339ce84413136/packaging-20.7-py2.py3-none-any.whl
Collecting colorama; sys_platform == "win32"
Downloading https://files.pythonhosted.org/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl
Collecting atomicwrites>=1.0; sys_platform == "win32"
Downloading https://files.pythonhosted.org/packages/2c/a0/da5f49008ec6e9a658dbf5d7310a4debd397bce0b4db03cf8a410066bb87/atomicwrites-1.4.0-py2.py3-none-any.whl
Collecting py>=1.8.2
Downloading https://files.pythonhosted.org/packages/68/0f/41a43535b52a81e4f29e420a151032d26f08b62206840c48d14b70e53376/py-1.9.0-py2.py3-none-any.whl (99kB)
Collecting pluggy<1.0,>=0.12
Downloading https://files.pythonhosted.org/packages/a0/28/85c7aa31b80d150b772fbe4a229487bc6644da9ccb7e427dd8cc60cb8a62/pluggy-0.13.1-py2.py3-none-any.whl
Collecting pycparser
Downloading https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl (112kB)
Collecting pyparsing>=2.0.2
Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67kB)
Installing collected packages: brotli, deflate, numpy, tqdm, pycparser, cffi, zstandard, attrs, iniconfig, toml, pyparsing, packaging, colorama, atomicwrites, py, pluggy, pytest
WARNING: The script f2py.exe is installed in 'C:\Python38-x64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script tqdm.exe is installed in 'C:\Python38-x64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts py.test.exe and pytest.exe are installed in 'C:\Python38-x64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed atomicwrites-1.4.0 attrs-20.3.0 brotli-1.0.9 cffi-1.14.4 colorama-0.4.4 deflate-0.2.0 iniconfig-1.1.1 numpy-1.19.4 packaging-20.7 pluggy-0.13.1 py-1.9.0 pycparser-2.20 pyparsing-2.4.7 pytest-6.1.2 toml-0.10.2 tqdm-4.54.1 zstandard-0.14.1
WARNING: You are using pip version 19.3.1; however, version 20.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
%PYTHON%\python.exe -m pip install -e .
Obtaining file:///C:/projects/mapbuffer
Requirement already satisfied: brotli in c:\python38-x64\lib\site-packages (from mapbuffer==0.2.1.dev5) (1.0.9)
Requirement already satisfied: deflate>=0.2.0 in c:\python38-x64\lib\site-packages (from mapbuffer==0.2.1.dev5) (0.2.0)
Requirement already satisfied: numpy in c:\python38-x64\lib\site-packages (from mapbuffer==0.2.1.dev5) (1.19.4)
Requirement already satisfied: tqdm in c:\python38-x64\lib\site-packages (from mapbuffer==0.2.1.dev5) (4.54.1)
Requirement already satisfied: zstandard in c:\python38-x64\lib\site-packages (from mapbuffer==0.2.1.dev5) (0.14.1)
Requirement already satisfied: cffi>=1.11 in c:\python38-x64\lib\site-packages (from zstandard->mapbuffer==0.2.1.dev5) (1.14.4)
Requirement already satisfied: pycparser in c:\python38-x64\lib\site-packages (from cffi>=1.11->zstandard->mapbuffer==0.2.1.dev5) (2.20)
Installing collected packages: mapbuffer
Found existing installation: mapbuffer 0.2.1.dev5
Can't uninstall 'mapbuffer'. No files were found to uninstall.
Running setup.py develop for mapbuffer
Successfully installed mapbuffer
WARNING: You are using pip version 19.3.1; however, version 20.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
%PYTHON%\python.exe -m pytest -v -x automated_test.py
============================= test session starts =============================
platform win32 -- Python 3.8.0, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- C:\Python38-x64\python.exe
cachedir: .pytest_cache
rootdir: C:\projects\mapbuffer
collecting ... collected 0 items / 1 error
=================================== ERRORS ====================================
_____________________ ERROR collecting automated_test.py ______________________
ImportError while importing test module 'C:\projects\mapbuffer\automated_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
C:\Python38-x64\lib\importlib\__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
automated_test.py:3: in <module>
from mapbuffer import MapBuffer, HEADER_LENGTH
mapbuffer\__init__.py:30: in <module>
from .mapbuffer import MapBuffer, HEADER_LENGTH, MAGIC_NUMBERS, FORMAT_VERSION
mapbuffer\mapbuffer.py:3: in <module>
from . import compression
mapbuffer\compression.py:5: in <module>
import deflate
E ModuleNotFoundError: No module named 'deflate'
=========================== short test summary info ===========================
ERROR automated_test.py
Thanks again for deflate! It's a really great module. ^_^
My code is located here if you'd like to look: https://github.com/seung-lab/mapbuffer
I don't have access to a Windows machine so I'm not sure what else I can do.
Will
> from deflate import DeflateError
E AttributeError: module 'deflate' has no attribute 'DeflateError'
see #12.
Hello,
I am receiving this error:
file = deflate.gzip_decompress(file.read())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
deflate.DeflateError: Decompression failed
Reproduce:
pip install deflate
Here is the code:
with open(file_path, "rb") as file:
file = deflate.gzip_decompress(file.read())
file = BytesIO(file)
The Python version is 3.12 and Ubuntu 22.04.
File is .gz
.
Thank you.
linux seems good, macOS (x64, Intel) mediocre, macOS (M1, Apple Silicon) the worst.
See there: #21
TODO: move insights from there to issues (guess best place is not here, but in libdeflate
's issue tracker.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.