dcwatson / deflate Goto Github PK

View Code? Open in Web Editor NEW

25.0 25.0 6.0 100 KB

Python extension wrapper for libdeflate.

License: MIT License

C 54.92% Python 44.19% Makefile 0.88%

deflate's People

Contributors

Stargazers

Watchers

Forkers

kylebarron seung-lab thomaswaldmann bket maarten-k drdub

deflate's Issues

CI via github actions?

we could easily run the tests (see #15) via github actions, so that they are automatically executed for any commit / pull request.

would you like a PR for that?

Windows support

I managed to build deflate under Windows with MSVC. I'm not an Python extensions expert so I'll just leave some pointers in case you decide to add official Windows support.

I had to make the following changes:

setup.py:

result = subprocess.run(["nmake", "/f", "Makefile.msc", "/a"], cwd=libdeflate_dir)
...
"deflate", sources=["deflate.c"], extra_objects=["libdeflate/libdeflatestatic.lib"]

Furthermore, MSVC's compiler breaks on:

deflate.c(72): error C2036: 'void *': unknown size

I'm not sure what the correct fix is, but I changed L72 to

bytes = (uint8_t *)(data.buf) + (data.len - 4);

to get it to compile.

Add `libdeflate_zlib` bindings

libdeflate exposes both libdeflate_gzip_compress and libdeflate_zlib_compress, but you only currently wrap the gzip one. Any plans to also wrap the zlib one?

Is this as simple as replacing libdeflate_gzip_compress with libdeflate_zlib_compress? Would seem straightforward (if verbose) to copy the bindings again for zlib?

segmentation fault using deflate_decompress

I've got a segmentation fault using the function deflate_decompress. Here is the test used:

(.env) @:~/wkpy/test$ python 
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import base64
>>> import deflate
>>> tt1 = "Sy1LzNFQt7dT10uvKs1Lzs8tKEotLtZIr8rMS8tJLEnVSEosTjUziU9JT\x635PSdUoLikqSi3TU\x43kuKTHQ\x42\x41Fr\x41\x41\x3d\x3d"
>>> tt1_decoded = base64.b64decode( tt1 )
>>> deflate.gzip_decompress( tt1_decoded )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
deflate.DeflateError: Invalid gzip data.
>>> deflate.deflate_decompress( tt1_decoded )
Segmentation fault (core dumped)

I was trying to decompress a file found in a php file, apparently compressed with the gzdeflate() function.

Hi again! Sorry to bug you. I was compiling a module against Windows 64-bit with Python 3.6, 3.7, and 3.8 on AppVeyor that includes deflate. Deflate appeared to successfully install, but was not found as a module. Here's my output:

uild started
git clone -q --branch=main https://github.com/seung-lab/mapbuffer.git C:\projects\mapbuffer
git checkout -qf 8d71424c63cb13c9babbdf307ff2d2ca6e1e4d31
Running Install scripts
%PYTHON%\python.exe -m pip install wheel
Collecting wheel
  Downloading https://files.pythonhosted.org/packages/c9/0b/e0fd299d93cd9331657f415085a4956422959897b333e3791dde40bd711d/wheel-0.36.1-py2.py3-none-any.whl
Installing collected packages: wheel
  WARNING: The script wheel.exe is installed in 'C:\Python38-x64\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed wheel-0.36.1
WARNING: You are using pip version 19.3.1; however, version 20.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
%PYTHON%\python.exe -m pip install -r requirements.txt -r requirements_dev.txt
Collecting brotli
  Downloading https://files.pythonhosted.org/packages/63/29/1b104b5915e61d9f7443889657d93937b7e0b33b331609b82693547934a0/Brotli-1.0.9-cp38-cp38-win_amd64.whl (365kB)
Collecting deflate>=0.2.0
  Downloading https://files.pythonhosted.org/packages/c2/42/49d6f7eecca4768e045b5bf25c8b7f650ffd88b5c0d927c5833161c496c2/deflate-0.2.0-cp38-cp38-win_amd64.whl
Collecting numpy
  Downloading https://files.pythonhosted.org/packages/40/db/5060f18b0116f00ee73f8365efc9c95bd5496946290b0e7c97b6ee89dffe/numpy-1.19.4-cp38-cp38-win_amd64.whl (13.0MB)
Collecting tqdm
  Downloading https://files.pythonhosted.org/packages/8a/54/115f0c28a61d56674c3a5e05c46d6c3523ad196e1dcd3e2d8b119026df36/tqdm-4.54.1-py2.py3-none-any.whl (69kB)
Collecting zstandard
  Downloading https://files.pythonhosted.org/packages/1d/60/888086d0239520c087da33ca2a2f1c185b24ce1d882ba68abbd8787c7884/zstandard-0.14.1-cp38-cp38-win_amd64.whl (551kB)
Collecting pytest
  Downloading https://files.pythonhosted.org/packages/b1/ee/53945d50284906adb1e613fabf2e1b8b25926e8676854bb25b93564c0ce7/pytest-6.1.2-py3-none-any.whl (272kB)
Collecting cffi>=1.11
  Downloading https://files.pythonhosted.org/packages/f6/d4/bf5516922cc914174a936e7094d5f136ee4178b3509069dd3da8e3040a1a/cffi-1.14.4-cp38-cp38-win_amd64.whl (179kB)
Collecting attrs>=17.4.0
  Downloading https://files.pythonhosted.org/packages/c3/aa/cb45262569fcc047bf070b5de61813724d6726db83259222cd7b4c79821a/attrs-20.3.0-py2.py3-none-any.whl (49kB)
Collecting iniconfig
  Downloading https://files.pythonhosted.org/packages/9b/dd/b3c12c6d707058fa947864b67f0c4e0c39ef8610988d7baea9578f3c48f3/iniconfig-1.1.1-py2.py3-none-any.whl
Collecting toml
  Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl
Collecting packaging
  Downloading https://files.pythonhosted.org/packages/28/87/8edcf555adaf60d053ead881bc056079e29319b643ca710339ce84413136/packaging-20.7-py2.py3-none-any.whl
Collecting colorama; sys_platform == "win32"
  Downloading https://files.pythonhosted.org/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl
Collecting atomicwrites>=1.0; sys_platform == "win32"
  Downloading https://files.pythonhosted.org/packages/2c/a0/da5f49008ec6e9a658dbf5d7310a4debd397bce0b4db03cf8a410066bb87/atomicwrites-1.4.0-py2.py3-none-any.whl
Collecting py>=1.8.2
  Downloading https://files.pythonhosted.org/packages/68/0f/41a43535b52a81e4f29e420a151032d26f08b62206840c48d14b70e53376/py-1.9.0-py2.py3-none-any.whl (99kB)
Collecting pluggy<1.0,>=0.12
  Downloading https://files.pythonhosted.org/packages/a0/28/85c7aa31b80d150b772fbe4a229487bc6644da9ccb7e427dd8cc60cb8a62/pluggy-0.13.1-py2.py3-none-any.whl
Collecting pycparser
  Downloading https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl (112kB)
Collecting pyparsing>=2.0.2
  Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67kB)
Installing collected packages: brotli, deflate, numpy, tqdm, pycparser, cffi, zstandard, attrs, iniconfig, toml, pyparsing, packaging, colorama, atomicwrites, py, pluggy, pytest
  WARNING: The script f2py.exe is installed in 'C:\Python38-x64\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script tqdm.exe is installed in 'C:\Python38-x64\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts py.test.exe and pytest.exe are installed in 'C:\Python38-x64\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed atomicwrites-1.4.0 attrs-20.3.0 brotli-1.0.9 cffi-1.14.4 colorama-0.4.4 deflate-0.2.0 iniconfig-1.1.1 numpy-1.19.4 packaging-20.7 pluggy-0.13.1 py-1.9.0 pycparser-2.20 pyparsing-2.4.7 pytest-6.1.2 toml-0.10.2 tqdm-4.54.1 zstandard-0.14.1
WARNING: You are using pip version 19.3.1; however, version 20.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
%PYTHON%\python.exe -m pip install -e .
Obtaining file:///C:/projects/mapbuffer
Requirement already satisfied: brotli in c:\python38-x64\lib\site-packages (from mapbuffer==0.2.1.dev5) (1.0.9)
Requirement already satisfied: deflate>=0.2.0 in c:\python38-x64\lib\site-packages (from mapbuffer==0.2.1.dev5) (0.2.0)
Requirement already satisfied: numpy in c:\python38-x64\lib\site-packages (from mapbuffer==0.2.1.dev5) (1.19.4)
Requirement already satisfied: tqdm in c:\python38-x64\lib\site-packages (from mapbuffer==0.2.1.dev5) (4.54.1)
Requirement already satisfied: zstandard in c:\python38-x64\lib\site-packages (from mapbuffer==0.2.1.dev5) (0.14.1)
Requirement already satisfied: cffi>=1.11 in c:\python38-x64\lib\site-packages (from zstandard->mapbuffer==0.2.1.dev5) (1.14.4)
Requirement already satisfied: pycparser in c:\python38-x64\lib\site-packages (from cffi>=1.11->zstandard->mapbuffer==0.2.1.dev5) (2.20)
Installing collected packages: mapbuffer
  Found existing installation: mapbuffer 0.2.1.dev5
    Can't uninstall 'mapbuffer'. No files were found to uninstall.
  Running setup.py develop for mapbuffer
Successfully installed mapbuffer
WARNING: You are using pip version 19.3.1; however, version 20.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
%PYTHON%\python.exe -m pytest -v -x automated_test.py
============================= test session starts =============================
platform win32 -- Python 3.8.0, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- C:\Python38-x64\python.exe
cachedir: .pytest_cache
rootdir: C:\projects\mapbuffer
collecting ... collected 0 items / 1 error
=================================== ERRORS ====================================
_____________________ ERROR collecting automated_test.py ______________________
ImportError while importing test module 'C:\projects\mapbuffer\automated_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
C:\Python38-x64\lib\importlib\__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
automated_test.py:3: in <module>
    from mapbuffer import MapBuffer, HEADER_LENGTH
mapbuffer\__init__.py:30: in <module>
    from .mapbuffer import MapBuffer, HEADER_LENGTH, MAGIC_NUMBERS, FORMAT_VERSION
mapbuffer\mapbuffer.py:3: in <module>
    from . import compression
mapbuffer\compression.py:5: in <module>
    import deflate
E   ModuleNotFoundError: No module named 'deflate'
=========================== short test summary info ===========================
ERROR automated_test.py

Thanks again for deflate! It's a really great module. ^_^

My code is located here if you'd like to look: https://github.com/seung-lab/mapbuffer

I don't have access to a Windows machine so I'm not sure what else I can do.

Will

Upgrade to libdeflate 1.18

Please stay up to date with the latest version of libdeflate so that people don't run into issues that have already been fixed. Thanks!

Add libdeflate gunzip logic to handle gzipped files which decompress to files larger than 4G or which contain multiple concatenated gzip files.

Some gzip files have 0 for the decompressed gzip length or contain multiple gzip files concatenated together so the size hint is only for the last gzip "file" instead of for the whole file.

https://github.com/ebiggers/libdeflate/blob/master/programs/gzip.c#L206-L273

update submodule to v1.10?

i've seen there is a newer release, maybe we should use it here?

AttributeError: module 'deflate' has no attribute 'DeflateError'

> from deflate import DeflateError
E       AttributeError: module 'deflate' has no attribute 'DeflateError'

Hi from python-isal

Hi, I work on python-isal, which wraps ISA-L. It also aims to accelerate compression/decompression and it supports streaming features.

Unfortunately ISA-L only works well on x86-64 (Intel, AMD) so it is much more limited than deflate in that respect.

Given that you probably work on this library because of some compression/decompression needs, I wanted to let you know about python-isal. Also I wanted to say hi, as another coder working on python bindings for a deflate-compatible compression library.

add git tags?

it would be nice if the changesets corresponding to a pypi release would be tagged like e.g. 0.3.0.

new release?

is there something left to do for a new pypi release which we can help with @dcwatson?

guess we can't do much about #23 because that is in libdeflate project scope, not in this project's scope (besides maybe doing an update of the subrepo later, IF relevant tuning happened there).

columns discrepancy

.clang-format says 88 columns
setup.cfg says 120 columns (i set that before noticing the value in .clang-format)

guess we want to adjust so both are the same.

Memory Leak

Hi! I think I found the existence of a memory leak somewhere in the library, but I haven't established where yet. I only know that memory snapshots indicated that deflate.gzip_decompress was being called and the memory was growing, memory profile plots were spiking, and when I uninstalled deflate and used gzip the problem disappeared.

I tried to reproduce the issue in a simpler environment with deflate isolated and was not successful.

Deflate is called here: https://github.com/seung-lab/cloud-files/blob/master/cloudfiles/compression.py#L184

Memory snapshots:

/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=66.4 KiB, count=939, average=72 B
/home/miniconda3/envs/ig/lib/python3.8/threading.py:244: size=63.8 KiB, count=217, average=301 B
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/_helpers.py:229: size=54.0 KiB, count=108, average=512 B
/home/miniconda3/envs/ig/lib/python3.8/email/message.py:484: size=41.0 KiB, count=674, average=62 B
Extracting Locations:   0%|▍                                                                                                                    | 500/143468 [00:04<19:15, 123.69it/s]/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=83.5 MiB, count=501, average=171 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=21.3 MiB, count=1000, average=21.8 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/_helpers.py:229: size=92.5 KiB, count=185, average=512 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=86.2 KiB, count=1248, average=71 B
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=81.3 KiB, count=685, average=122 B
Extracting Locations:   1%|▊                                                                                                                   | 1000/143468 [00:10<23:06, 102.78it/s]/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=225 MiB, count=501, average=459 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=80.0 MiB, count=1500, average=54.6 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/_helpers.py:229: size=80.5 KiB, count=161, average=512 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=79.8 KiB, count=1147, average=71 B
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=78.0 KiB, count=661, average=121 B
Extracting Locations:   1%|█▏                                                                                                                   | 1500/143468 [00:21<30:54, 76.56it/s]/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=199 MiB, count=501, average=408 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=132 MiB, count=2000, average=67.7 KiB
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=73.2 KiB, count=626, average=120 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=70.9 KiB, count=1008, average=72 B
/home/miniconda3/envs/ig/lib/python3.8/tracemalloc.py:532: size=68.5 KiB, count=1342, average=52 B
Extracting Locations:   1%|█▋                                                                                                                   | 2000/143468 [00:30<35:20, 66.73it/s]/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=203 MiB, count=501, average=415 KiB
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=185 MiB, count=2500, average=75.9 KiB
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=75.8 KiB, count=644, average=120 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=75.2 KiB, count=1077, average=72 B
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/_helpers.py:229: size=72.0 KiB, count=144, average=512 B
Extracting Locations:   2%|██                                                                                                                   | 2500/143468 [00:40<38:12, 61.50it/s]/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=241 MiB, count=3000, average=82.2 KiB
/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=214 MiB, count=501, average=437 KiB
/mnt/data02/ws9/cloud-volume/cloudvolume/datasource/precomputed/spatial_index.py:166: size=883 KiB, count=5674, average=159 B
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=75.6 KiB, count=644, average=120 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=75.3 KiB, count=1078, average=72 B
Extracting Locations:   2%|██▍                                                                                                                  | 3000/143468 [00:50<40:38, 57.60it/s]/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=292 MiB, count=3500, average=85.5 KiB
/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=198 MiB, count=501, average=405 KiB
/mnt/data02/ws9/cloud-volume/cloudvolume/datasource/precomputed/spatial_index.py:166: size=927 KiB, count=6432, average=148 B
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=70.3 KiB, count=604, average=119 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=65.2 KiB, count=920, average=73 B
Extracting Locations:   2%|██▊                                                                                                                  | 3500/143468 [01:00<41:49, 55.78it/s]/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/blob.py:1252: size=345 MiB, count=4000, average=88.4 KiB
/mnt/data02/ws9/cloud-files/cloudfiles/compression.py:184: size=210 MiB, count=501, average=428 KiB
/home/miniconda3/envs/ig/lib/python3.8/posixpath.py:88: size=75.3 KiB, count=632, average=122 B
/home/miniconda3/envs/ig/lib/python3.8/email/_policybase.py:303: size=72.1 KiB, count=1027, average=72 B
/home/miniconda3/envs/ig/lib/python3.8/site-packages/google/cloud/storage/_helpers.py:229: size=66.0 KiB, count=132, average=512 B

With deflate installed (black) vs uninstalled (red,blue):

I'll take a look through the code and see if I can find anything. One thing to keep in mind is that it's being executed in a thread: https://github.com/seung-lab/cloud-files/blob/master/cloudfiles/cloudfiles.py#L177

Python 3.12 Wheels

Hi!

Happy holidays!

I've been just doing a routine update of all my libraries and dependency audit for Python 3.12. Would it be possible to get Python 3.12 wheels uploaded?

Thank you so much for all the work you've put into this!

Will

performance issues

linux seems good, macOS (x64, Intel) mediocre, macOS (M1, Apple Silicon) the worst.

See there: #21

TODO: move insights from there to issues (guess best place is not here, but in libdeflate's issue tracker.

Can this library be used for streaming decompression?

Thanks for writing this wrapper, is it possible to use it to stream decompress chunked data? Either through a pipe or streaming HTTP response?

Thanks again.

add deflate.crc32 api?

Hi Dan,

greetings from borgbackup development!

We currently discuss about getting rid of some code we have bundled and currently there is only crc32 code left.

We were searching where we could get some fast crc32 from and found libdeflate and your python wrapper for it.

Discussion: borgbackup/borg#6316

Do you think a crc32 api could be added, so we could easily use it?

macos "build wheels" ran for 5h when i cancelled it

https://github.com/dcwatson/deflate/runs/5433169119?check_suite_focus=true

this was running for the 0.3.0 tag after I tagged it.

do we need a timeout there?

fix pypy support?

after adding testing to the automated wheel building running via github actions, i noticed that the pypy wheel does not test ok. likely no new issue, just nobody noticed yet / complained yet.

when this issue is fixed, the pypy wheel building can be re-enabled.

see there:

c2ebdba
3887969
#36 normal github actions CI with pypy-3.8 also fails (so it is a general issue, not just with the wheel)

tests?

would be cool to have some tests for the code, like:

roundtrip (test decompress(compress(data)) == data for misc. data values)
test api, exception raising
pytest is a nice testing framework

todo?:

"speed test" that always succeeds and outputs speed as a side effect (maybe also in comparison to gzip / zlib python stdlib code)
pytest-benchmark is nice for benchmarks.

Can't be done / out of scope:

zlib compatibility? (test deflate_compress(data) == gzip_compress(data) for misc. data values)
test the different internal implementations, if possible

Error Building with Miniconda Python 3.8

Thank you for this wonderful library! While I was experimenting with it in miniconda with python3.8 I ran into an error.

The line that fails appears to be part of Anaconda and not part of the usual compilation chain and relates to libdeflate.a being a static library.

The problem seems to exist on Ubuntu 16.04. It compiles fine on my macbook.

 gcc -pthread -shared -B $MY_DIRECTORY/compiler_compat -$LMY_DIRECTORY/lib -Wl,-rpath=$MY_DIRECTORY/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/deflate.o libdeflate/libdeflate.a -o /mnt/data02/ws9/deflate/deflate.cpython-38-x86_64-linux-gnu.so

Full trace:

    ERROR: Command errored out with exit status 1:
     command: $MY_DIRECTORY/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/mnt/data02/ws9/deflate/setup.py'"'"'; __file__='"'"'/mnt/data02/ws9/deflate/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /mnt/data02/ws9/deflate/
    Complete output (51 lines):
    running develop
    running egg_info
    writing deflate.egg-info/PKG-INFO
    writing dependency_links to deflate.egg-info/dependency_links.txt
    writing top-level names to deflate.egg-info/top_level.txt
    reading manifest file 'deflate.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'deflate.egg-info/SOURCES.txt'
    running build_ext
      CC       lib/deflate_decompress.o
      CC       lib/utils.o
      CC       lib/arm/cpu_features.o
      CC       lib/x86/cpu_features.o
      CC       lib/deflate_compress.o
      CC       lib/adler32.o
      CC       lib/zlib_decompress.o
      CC       lib/zlib_compress.o
      CC       lib/crc32.o
      CC       lib/gzip_decompress.o
      CC       lib/gzip_compress.o
      AR       libdeflate.a
      CC       lib/deflate_decompress.shlib.o
      CC       lib/utils.shlib.o
      CC       lib/arm/cpu_features.shlib.o
      CC       lib/x86/cpu_features.shlib.o
      CC       lib/deflate_compress.shlib.o
      CC       lib/adler32.shlib.o
      CC       lib/zlib_decompress.shlib.o
      CC       lib/zlib_compress.shlib.o
      CC       lib/crc32.shlib.o
      CC       lib/gzip_decompress.shlib.o
      CC       lib/gzip_compress.shlib.o
      CCLD     libdeflate.so.0
      LN       libdeflate.so
      GEN      programs/config.h
      CC       programs/gzip.o
      CC       programs/prog_util.o
      CC       programs/tgetopt.o
      CCLD     gzip
      LN       gunzip
    building 'deflate' extension
    creating build
    creating build/temp.linux-x86_64-3.8
    gcc -pthread -B $MY_DIRECTORY/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -$IMY_DIRECTORY/include/python3.8 -c deflate.c -o build/temp.linux-x86_64-3.8/deflate.o
    gcc -pthread -shared -B $MY_DIRECTORY/compiler_compat -$LMY_DIRECTORY/lib -Wl,-rpath=$MY_DIRECTORY/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/deflate.o libdeflate/libdeflate.a -o /mnt/data02/ws9/deflate/deflate.cpython-38-x86_64-linux-gnu.so
    $MY_DIRECTORY/compiler_compat/ld: libdeflate/libdeflate.a(deflate_decompress.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
    $MY_DIRECTORY/compiler_compat/ld: libdeflate/libdeflate.a(deflate_compress.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
    $MY_DIRECTORY/compiler_compat/ld: libdeflate/libdeflate.a(crc32.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
    $MY_DIRECTORY/compiler_compat/ld: final link failed: nonrepresentable section on output
    collect2: error: ld returned 1 exit status
    error: command 'gcc' failed with exit status 1
    ----------------------------------------