pypa / auditwheel Goto Github PK

View Code? Open in Web Editor NEW

421.0 421.0 138.0 1.12 MB

Auditing and relabeling cross-distribution Linux wheels.

License: Other

Python 95.50% Fortran 0.18% C 2.94% C++ 0.84% Makefile 0.32% Cython 0.21%

auditwheel's People

Contributors

Stargazers

Watchers

Forkers

rmcgibbo hainm pvanderlinden ogrisel anlambert pombredanne ehashman soltanmm vitaly-krugl grazies mathmagique xhochy tdsmith pkittenis lbianc ppc64 jakirkham robind42 hredestig szilveszter mckaymatt reaperhulk alex sbidoul kleopatra999 dvarrazzo jgillis mfornace yunstanford markrwilliams dannyzed gukoff ravshanof bmoyles wtolson mayeut cliffwoolley ofek louhibi hugovk aaronchall anthrotype kaikulimu mariand012 isaac-philip atombrella andreysmelter daa thomaslima yuhonghong7035 di kirill888 takluyver lkollar jcfr 0x6b7966 kpostoffice dsteinmo veblush westurner czaki wdirons delarosatrevin ferdonline sgillies durbin86 sneakers-the-rat rcoup fbrennen lpsinger wiggin15 global-localhost global19 global19-atlassian-net stevenans985900 bdice isabella232 johnwparent tlc-pack davhau polochon-street baby636 rossant oneflow-inc terrorizer1980 deworsky29 qksidmx hroncok sunnycase encukou fbxiang zsimic tipabu externalrepositories luminartech f3flight martinrenou jlaine python-repository-hub eoyilmaz

auditwheel's Issues

ValueError in versioned symbols

When compiling a manylinux1 wheel against OpenSSL 1.1.0 the following exception is raised:

  File "/usr/local/bin/auditwheel", line 11, in <module>
    sys.exit(main())
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/main.py", line 49, in main
    rval = args.func(args, p)
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/main_repair.py", line 60, in execute
    wheel_abi = analyze_wheel_abi(args.WHEEL_FILE)
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/wheel_abi.py", line 82, in analyze_wheel_abi
    symbol_policy = versioned_symbols_policy(versioned_symbols)
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/policy/versioned_symbols.py", line 28, in versioned_symbols_policy
    sym_name, _ = symbol.split("_", 2)
ValueError: too many values to unpack (expected 2)

Looking in PDB the issue is that the symbol is expected to look something like GLIBC_2.2.5, but OpenSSL 1.1.0 is OPENSSL_1_1_0, which results in too many values.

Undefined name 'src_name' in auditwheel/repair.py

Should this be src_path instead of src_name?

flake8 testing of https://github.com/pypa/auditwheel on Python 3.6.3

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./auditwheel/repair.py:120:22: F821 undefined name 'src_name'
        new_soname = src_name
                     ^

auditwheel repair should not fail on pure Python wheels

... IMHO it should just do nothing in this case.
I experienced this while using the manylinux docker images on Travis here https://travis-ci.org/pombreda/thirdparty-manylinux/jobs/152792412#L697
The auditwheel step fails because a six wheels had been pulled in

Support multiple wheel files in repair command

Currently auditwheel repair command supports only one WHEEL_FILE argument, I thought it would be nice if it will be possible to give multiple wheels to repair, what do you think? It will be useful in my use case when I build a set of wheels some of which should be repaired. And if you consider it to be useful addition, I'll submit a patch.

README here and on PyPI points to the old docker images

auditwheel not finding shared libraries

I've been playing with manylinux builds and can't seem to get auditwheel to ever find shared libraries and fix them.

For example, I've tried the following by cloning https://github.com/pypa/python-manylinux-demo , navigating to its root directory and running

docker run -v `pwd`:/io -it --rm quay.io/pypa/manylinux1_x86_64 /bin/bash

From inside the container, I then build the wheels via

/io/travis/build-wheels.sh

Everything completes successfully, but the auditwheel repair doesn't seem to have packaged anything up and the wheels have no .libs directory containing atlas.

From inside the container, running auditwheel -vv show python_manylinux_demo-1.0-cp27-cp27mu-linux_x86_64.whl gives the following:

DEBUG:auditwheel.wheel_abi:{}
DEBUG:auditwheel.policy.versioned_symbols:Required symbol versions: {}
INFO:auditwheel.wheel_abi:{
    "manylinux1_x86_64": {
        "priority": 100,
        "libs": {}
    },
    "linux_x86_64": {
        "priority": 0,
        "libs": {}
    }
}
DEBUG:auditwheel.wheel_abi:external referene info
DEBUG:auditwheel.wheel_abi:{
    "manylinux1_x86_64": {
        "priority": 100,
        "libs": {}
    },
    "linux_x86_64": {
        "priority": 0,
        "libs": {}
    }
}

python_manylinux_demo-1.0-cp27-cp27mu-linux_x86_64.whl is consistent
with the following platform tag: "manylinux1_x86_64".

The wheel references no external versioned symbols from system-
provided shared libraries.

The wheel requires no external shared libraries! :)

but unzipping that wheel and runnning ldd tells a different story

[root@1965501e43ea pymanylinuxdemo]# ldd extension.so
        linux-vdso.so.1 =>  (0x00007ffc52bd8000)
        libcblas.so.3 => /usr/lib64/atlas/libcblas.so.3 (0x00007fdcd467e000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdcd4461000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fdcd4108000)
        libatlas.so.3 => /usr/lib64/atlas/libatlas.so.3 (0x00007fdcd3813000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fdcd358f000)
        /lib64/ld-linux-x86-64.so.2 (0x00005619467b3000)

Any help would be greatly appreciated! I'm experiencing the same thing across the board with other builds I've been trying to manylinuxize.

max_versioned_symbol breaks when looking at symbols it doesn't need to

After building some python-kadmin wheels on the manylinux docker image, I encountered an exception when running auditwheel on the wheels:

$ auditwheel -vv repair wheelhouse/python_kadmin-0.1.1-cp26-cp26m-linux_x86_64.whl -w /io/wheelhouse
...<snip>
Traceback (most recent call last):
  File "/usr/local/bin/auditwheel", line 11, in <module>
    sys.exit(main())
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main.py", line 49, in main
    rval = args.func(args, p)
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main_repair.py", line 60, in execute
    wheel_abi = analyze_wheel_abi(args.WHEEL_FILE)
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/wheel_abi.py", line 59, in analyze_wheel_abi
    get_wheel_elfdata(wheel_fn)
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/wheel_abi.py", line 48, in get_wheel_elfdata
    max_versioned_symbol(versioned_symbols), uses_ucs2_symbols)
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/policy/versioned_symbols.py", line 22, in max_versioned_symbol
    set_if_greater(max_required_ver, name, Version(ver))
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/pip/_vendor/distlib/version.py", line 32, in __init__
    self._parts = parts = self.parse(s)
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/pip/_vendor/distlib/version.py", line 273, in parse
    result = _normalized_key(s)
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/pip/_vendor/distlib/version.py", line 194, in _pep_440_key
    raise UnsupportedVersionError('Not a valid version: %s' % s)

pip._vendor.distlib.version.UnsupportedVersionError: Not a valid version: MIT

(full Travis log)

Taking a look at the version symbols inside krb5, we find:

$ readelf -V ./lib/krb5/libkrb5.so
...<snip>
Version definition section '.gnu.version_d' contains 3 entries:
  Addr: 0x000000000000bcc0  Offset: 0x00bcc0  Link: 4 (.dynstr)  000000: Rev: 1  Flags: BASE   Index: 1  Cnt: 1  Name: libkrb5.so.3
  0x001c: Rev: 1  Flags: none  Index: 2  Cnt: 1  Name: krb5_3_MIT
  0x0038: Rev: 1  Flags: none  Index: 3  Cnt: 1  Name: HIDDEN

Version needs section '.gnu.version_r' contains 5 entries:
 Addr: 0x000000000000bd18  Offset: 0x00bd18  Link: 4 (.dynstr)
  000000: Version: 1  File: libcom_err.so.3  Cnt: 1
  0x0010:   Name: com_err_3_MIT  Flags: none  Version: 10
  0x0020: Version: 1  File: libresolv.so.2  Cnt: 2
  0x0030:   Name: GLIBC_2.2.5  Flags: none  Version: 13
  0x0040:   Name: GLIBC_2.9  Flags: none  Version: 9
  0x0050: Version: 1  File: libk5crypto.so.3  Cnt: 1
  0x0060:   Name: k5crypto_3_MIT  Flags: none  Version: 7
  0x0070: Version: 1  File: libkrb5support.so.0  Cnt: 1
  0x0080:   Name: krb5support_0_MIT  Flags: none  Version: 6
  0x0090: Version: 1  File: libc.so.6  Cnt: 7
  0x00a0:   Name: GLIBC_2.14  Flags: none  Version: 15
  0x00b0:   Name: GLIBC_2.8  Flags: none  Version: 14
  0x00c0:   Name: GLIBC_2.16  Flags: none  Version: 12
  0x00d0:   Name: GLIBC_2.4  Flags: none  Version: 11
  0x00e0:   Name: GLIBC_2.3.4  Flags: none  Version: 8
  0x00f0:   Name: GLIBC_2.2.5  Flags: none  Version: 5
  0x0100:   Name: GLIBC_2.3  Flags: none  Version: 4

So I believe the issue here is that we are attempting to determine the max symbol version on all included version symbols, even though we only care about the ones in policy.json (GLIBC, CXXABI, etc.). While there are conventions for the numeric versioning for the GLIBC etc. symbols, this is not generally true, which is why we're running into the issue with e.g. k5crypto_3_MIT having a version read as MIT. A fix would be to filter out any symbols we don't care about per policy before running max_versioned_symbol on them?

Full example; from building the initial wheel to repairing it using auditwheel

I was hoping to get a full blown example using the setup.py that built

cffi-1.5.0-cp35-cp35m-linux_x86_64.whl

I'm not sure how to include the .so files as dependencies when building the initial wheel i.e:

{
    "libc.so.6": "/lib64/libc-2.5.so",
    "libffi.so.5": "/usr/lib64/libffi.so.5.0.6",
    "libpthread.so.0": "/lib64/libpthread-2.5.so"
}

Thanks, Mike.

Implement manylinux/pulls/31

To implement pypa/manylinux#31:

Revert UCS2/UCS4 symbol detection
Remove existing special-casing of libpythonX.Y.so in whitelist, but add different special-casing that tells you to build with a --disable-shared python and/or uses patchelf --remove-needed.
Add mandate that py2 and py30-32 wheels contain an ABI tag other than 'none' when the platform tag is added.

But you should never have a wheel with a name like::

  PKG-VERSION-cp27-none-manylinux1_x86_64.whl
                  ^^^^ BAD! Don't do this!

Describe which symbols are too recent

Using the manylinux docker image, I get this error message:

auditwheel: error: cannot repair "wheelhouse/pygame-1.9.2.dev1-cp34-cp34m-linux_x86_64.whl" to "manylinux1_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.

It would be helpful to know which symbols it doesn't like!

Support repairing cffi PyPy extensions

CFFI extensions compiled for PyPy do not export 'init' + modname nor 'PyInit_' + modname symbols but instead they export '_cffi_pypyinit_' + modname and thus they are skipped by auditwheel. However actually auditwheel can repair them. As an example one may take a look at python-snappy - it uses cffi for PyPy. And proof for using _cffi_pypyinit_ prefix may be found at cffi/recompiler.py.

Auditwheel stripping rpath for relative paths

I'm developing a library that has several C++ extensions, some of which are linked internally. For each extension library that has a dependency, I add the relative path (with $ORIGIN) to the linked library, as such:

runtime_library_dirs.append("\$ORIGIN/%s" % <rel-path-between modA.so and modB.so>)

This gives me .so files with dependencies:

 0x0000000000000001 (NEEDED)             Shared library: [libptf77blas.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [libptcblas.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [libatlas.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [modA.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [modB.so]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/relpath/to/modA]

However, when using the auditwheel tool, the runpath is stripped and replaced with $ORIGIN/../../../.libs. As such, my library (here modA.so) can no longer be found:

 0x0000000000000001 (NEEDED)             Shared library: [libptf77blas-536e8f3e.so.3.0]
 0x0000000000000001 (NEEDED)             Shared library: [libptcblas-f940d3f1.so.3.0]
 0x0000000000000001 (NEEDED)             Shared library: [libatlas-dd66cbe8.so.3.0]
 0x0000000000000001 (NEEDED)             Shared library: [modA.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [modB.so]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../.libs]

It would be really nice if there was a way to have the runpath appended with the path to the grafted libraries, and not replaced. Is there some solution to this? Or, am I missing something?

auditwheel repair does not graft shared library dependencies

I don't know why, auditwheel does not graft the shared library dependencies to my manylinux wheel.
Below is the screen output.

[root@d52c3a246527 dist]# auditwheel -v show ecell-4.1.0-cp35-cp35m-linux_x86_64.whl
INFO:auditwheel.wheel_abi:processing: ecell4/bd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.lddtree:linker ldpaths: {'conf': ['/lib', '/lib64/', '/usr/lib', '/usr/lib64'], 'env': ['/opt/rh/devtoolset-2/root/usr/lib64', '/opt/rh/devtoolset-2/root/usr/lib', '/usr/local/lib64', '/usr/local/lib'], 'interp': []}
INFO:auditwheel.wheel_abi:processing: ecell4/core.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/egfrd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/gillespie.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/meso.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/ode.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/spatiocyte.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:{
    "linux_x86_64": {
        "libs": {
            "libm.so.6": "/lib64/libm-2.5.so",
            "libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
            "libhdf5_cpp.so.11": "/usr/local/lib/libhdf5_cpp.so.11.0.0",
            "libc.so.6": "/lib64/libc-2.5.so",
            "libhdf5.so.10": "/usr/local/lib/libhdf5.so.10.1.0",
            "librt.so.1": "/lib64/librt-2.5.so",
            "libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
            "libpthread.so.0": "/lib64/libpthread-2.5.so",
            "libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0",
            "libdl.so.2": "/lib64/libdl-2.5.so",
            "libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8"
        },
        "priority": 0
    },
    "manylinux1_x86_64": {
        "libs": {
            "libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
            "libhdf5_cpp.so.11": "/usr/local/lib/libhdf5_cpp.so.11.0.0",
            "libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0",
            "libhdf5.so.10": "/usr/local/lib/libhdf5.so.10.1.0"
        },
        "priority": 100
    }
}

ecell-4.1.0-cp35-cp35m-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.0'},
libpthread.so.0 with versions {'GLIBC_2.2.5'}, libc.so.6 with versions
{'GLIBC_2.2.5', 'GLIBC_2.4', 'GLIBC_2.3.4'}, libm.so.6 with versions
{'GLIBC_2.2.5'}, libstdc++.so.6 with versions {'GLIBCXX_3.4',
'CXXABI_1.3.1', 'CXXABI_1.3'}

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib64/libc-2.5.so",
    "libdl.so.2": "/lib64/libdl-2.5.so",
    "libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
    "libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
    "libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0",
    "libhdf5.so.10": "/usr/local/lib/libhdf5.so.10.1.0",
    "libhdf5_cpp.so.11": "/usr/local/lib/libhdf5_cpp.so.11.0.0",
    "libm.so.6": "/lib64/libm-2.5.so",
    "libpthread.so.0": "/lib64/libpthread-2.5.so",
    "librt.so.1": "/lib64/librt-2.5.so",
    "libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8"
}

In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:

libgsl.so.0, libgslcblas.so.0, libhdf5.so.10, libhdf5_cpp.so.11
[root@d52c3a246527 dist]# auditwheel -v repair ecell-4.1.0-cp35-cp35m-linux_x86_64.whl
Repairing ecell-4.1.0-cp35-cp35m-linux_x86_64.whl
INFO:auditwheel.wheel_abi:processing: ecell4/bd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.lddtree:linker ldpaths: {'conf': ['/lib', '/lib64/', '/usr/lib', '/usr/lib64'], 'env': ['/opt/rh/devtoolset-2/root/usr/lib64', '/opt/rh/devtoolset-2/root/usr/lib', '/usr/local/lib64', '/usr/local/lib'], 'interp': []}
INFO:auditwheel.wheel_abi:processing: ecell4/core.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/egfrd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/gillespie.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/meso.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/ode.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/spatiocyte.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:{
    "linux_x86_64": {
        "libs": {
            "libhdf5.so.10": "/usr/local/lib/libhdf5.so.10.1.0",
            "libpthread.so.0": "/lib64/libpthread-2.5.so",
            "libc.so.6": "/lib64/libc-2.5.so",
            "libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8",
            "libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
            "libm.so.6": "/lib64/libm-2.5.so",
            "libdl.so.2": "/lib64/libdl-2.5.so",
            "librt.so.1": "/lib64/librt-2.5.so",
            "libhdf5_cpp.so.11": "/usr/local/lib/libhdf5_cpp.so.11.0.0",
            "libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
            "libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0"
        },
        "priority": 0
    },
    "manylinux1_x86_64": {
        "libs": {
            "libhdf5_cpp.so.11": "/usr/local/lib/libhdf5_cpp.so.11.0.0",
            "libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
            "libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0",
            "libhdf5.so.10": "/usr/local/lib/libhdf5.so.10.1.0"
        },
        "priority": 100
    }
}
Grafting: /usr/local/lib/libhdf5_cpp.so.11.0.0 -> ecell4/.libs/libhdf5_cpp-44aff6ad.so.11.0.0
Setting RPATH: ecell4/.libs/libhdf5_cpp-44aff6ad.so.11.0.0 to "$ORIGIN/."
Grafting: /usr/lib64/libgsl.so.0.14.0 -> ecell4/.libs/libgsl-b885fd68.so.0.14.0
Grafting: /usr/lib64/libgslcblas.so.0.0.0 -> ecell4/.libs/libgslcblas-15b2d0fe.so.0.0.0
Grafting: /usr/local/lib/libhdf5.so.10.1.0 -> ecell4/.libs/libhdf5-669db1ee.so.10.1.0
Setting RPATH: ecell4/bd.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/core.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/egfrd.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/gillespie.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/meso.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/ode.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/spatiocyte.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Previous filename tags: linux_x86_64
New filename tags: manylinux1_x86_64
Previous WHEEL info tags: cp35-cp35m-linux_x86_64
New WHEEL info tags: cp35-cp35m-manylinux1_x86_64

Fixed-up wheel written to /root/ecell4/python/dist/wheelhouse/ecell-4.1.0-cp35-cp35m-manylinux1_x86_64.whl
[root@d52c3a246527 dist]# auditwheel -v show wheelhouse/ecell-4.1.0-cp35-cp35m-manylinux1_x86_64.whl
INFO:auditwheel.wheel_abi:processing: ecell4/core.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.lddtree:linker ldpaths: {'conf': ['/lib', '/lib64/', '/usr/lib', '/usr/lib64'], 'env': ['/opt/rh/devtoolset-2/root/usr/lib64', '/opt/rh/devtoolset-2/root/usr/lib', '/usr/local/lib64', '/usr/local/lib'], 'interp': []}
INFO:auditwheel.wheel_abi:processing: ecell4/gillespie.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/bd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/ode.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/spatiocyte.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/meso.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/egfrd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:{
    "linux_x86_64": {
        "libs": {
            "libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8",
            "libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
            "libpthread.so.0": "/lib64/libpthread-2.5.so",
            "libm.so.6": "/lib64/libm-2.5.so",
            "libdl.so.2": "/lib64/libdl-2.5.so",
            "librt.so.1": "/lib64/librt-2.5.so",
            "libc.so.6": "/lib64/libc-2.5.so"
        },
        "priority": 0
    },
    "manylinux1_x86_64": {
        "libs": {},
        "priority": 100
    }
}

ecell-4.1.0-cp35-cp35m-manylinux1_x86_64.whl is consistent with the
following platform tag: "manylinux1_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.0'},
libpthread.so.0 with versions {'GLIBC_2.2.5'}, libc.so.6 with versions
{'GLIBC_2.2.5', 'GLIBC_2.3.4', 'GLIBC_2.4'}, libm.so.6 with versions
{'GLIBC_2.2.5'}, libstdc++.so.6 with versions {'CXXABI_1.3.1',
'CXXABI_1.3', 'GLIBCXX_3.4'}

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib64/libc-2.5.so",
    "libdl.so.2": "/lib64/libdl-2.5.so",
    "libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
    "libm.so.6": "/lib64/libm-2.5.so",
    "libpthread.so.0": "/lib64/libpthread-2.5.so",
    "librt.so.1": "/lib64/librt-2.5.so",
    "libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8"
}

In my manylinux wheel, it seems that hdf5 and gsl are not grafted by auditwheel.
I would be grateful if you could give me some clues on this.

This can be reproduced with https://github.com/ecell/ecell4-manylinux/blob/v4.1.0/.travis.yml
I used the latest quay.io/pypa/manylinux1_x86_64 image to build my wheel.
The auditwheel version is

[root@d52c3a246527 ~]# auditwheel -V
auditwheel 1.5.0 installed at /opt/_internal/cpython-3.6.0/lib/python3.6/site-
packages (python 3.6)

When a single wheel contains multiple top-level packages, auditwheel creates separate .libs dirs for each of them

Bug originally filed by @lpsinger at pypa/manylinux#156:

If you have a package that contains multiple C extensions that depend on the same shared libraries, then auditwheel repair saves duplicate copies of all of the shared libraries. An example is shown below.

This makes for packages that are several times larger than necessary. Could auditwheel repair sweep up all of the dependencies and put them in a single .libs directory, rather than creating duplicates?

Example

$ tar -tf lalsuite-6.48-cp27-cp27m-manylinux1_x86_64.whl 
lalxml/git_version.py
lalxml/_lalxml.so
lalxml/__init__.py
lalxml/lalxml.py
lalxml/.libs/libgsl-b885fd68.so.0.14.0
lalxml/.libs/libhdf5_hl-04af5206.so.100.0.1
lalxml/.libs/libhdf5-db4e48f9.so.101.0.0
lalxml/.libs/liblalsupport-bd9338fc.so.11.0.0
lalxml/.libs/liblalxml-a864f854.so.2.0.1
lalxml/.libs/libxml2-8ed442f7.so.2.6.26
lalxml/.libs/libz-a147dcb0.so.1.2.3
lalxml/.libs/libfftw3-c8dcb815.so.3.2.4
lalxml/.libs/liblal-8b0d5843.so.14.0.0
lalxml/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalxml/.libs/libfftw3f-044fdcc5.so.3.2.4
lalmetaio/git_version.py
lalmetaio/lalmetaio.py
lalmetaio/_lalmetaio.so
lalmetaio/__init__.py
lalmetaio/.libs/libgsl-b885fd68.so.0.14.0
lalmetaio/.libs/libhdf5_hl-04af5206.so.100.0.1
lalmetaio/.libs/libhdf5-db4e48f9.so.101.0.0
lalmetaio/.libs/liblalsupport-bd9338fc.so.11.0.0
lalmetaio/.libs/liblalmetaio-f4e58690.so.5.0.1
lalmetaio/.libs/libz-a147dcb0.so.1.2.3
lalmetaio/.libs/libfftw3-c8dcb815.so.3.2.4
lalmetaio/.libs/liblal-8b0d5843.so.14.0.0
lalmetaio/.libs/libmetaio-c02fec4d.so.1.1.0
lalmetaio/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalmetaio/.libs/libfftw3f-044fdcc5.so.3.2.4
lal/_lal.so
lal/gpstime.py
lal/lal.py
lal/git_version.py
lal/__init__.py
lal/rate.py
lal/series.py
lal/.libs/libgsl-b885fd68.so.0.14.0
lal/.libs/libhdf5_hl-04af5206.so.100.0.1
lal/.libs/libhdf5-db4e48f9.so.101.0.0
lal/.libs/liblalsupport-bd9338fc.so.11.0.0
lal/.libs/libz-a147dcb0.so.1.2.3
lal/.libs/libfftw3-c8dcb815.so.3.2.4
lal/.libs/liblal-8b0d5843.so.14.0.0
lal/.libs/libgslcblas-15b2d0fe.so.0.0.0
lal/.libs/libfftw3f-044fdcc5.so.3.2.4
lal/utils/cache.py
lal/utils/__init__.py
lal/utils/series.py
lal/spectrum/distributions.py
lal/spectrum/averagespectrum.py
lal/spectrum/__init__.py
lalsuite-6.48.dist-info/WHEEL
lalsuite-6.48.dist-info/METADATA
lalsuite-6.48.dist-info/DESCRIPTION.rst
lalsuite-6.48.dist-info/metadata.json
lalsuite-6.48.dist-info/RECORD
lalsuite-6.48.dist-info/top_level.txt
lalsimulation/git_version.py
lalsimulation/__init__.py
lalsimulation/_lalsimulation.so
lalsimulation/lalsimulation.py
lalsimulation/.libs/libgsl-b885fd68.so.0.14.0
lalsimulation/.libs/libhdf5_hl-04af5206.so.100.0.1
lalsimulation/.libs/liblalsimulation-f35d2299.so.16.0.3
lalsimulation/.libs/libhdf5-db4e48f9.so.101.0.0
lalsimulation/.libs/liblalsupport-bd9338fc.so.11.0.0
lalsimulation/.libs/libz-a147dcb0.so.1.2.3
lalsimulation/.libs/libfftw3-c8dcb815.so.3.2.4
lalsimulation/.libs/liblal-8b0d5843.so.14.0.0
lalsimulation/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalsimulation/.libs/libfftw3f-044fdcc5.so.3.2.4
lalframe/git_version.py
lalframe/frread.py
lalframe/__init__.py
lalframe/lalframe.py
lalframe/_lalframe.so
lalframe/.libs/libgsl-b885fd68.so.0.14.0
lalframe/.libs/libhdf5_hl-04af5206.so.100.0.1
lalframe/.libs/libhdf5-db4e48f9.so.101.0.0
lalframe/.libs/liblalsupport-bd9338fc.so.11.0.0
lalframe/.libs/libFrame-0c783110.so.1.5.0
lalframe/.libs/liblalframe-2600de1c.so.10.0.2
lalframe/.libs/libz-a147dcb0.so.1.2.3
lalframe/.libs/libfftw3-c8dcb815.so.3.2.4
lalframe/.libs/liblal-8b0d5843.so.14.0.0
lalframe/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalframe/.libs/libfftw3f-044fdcc5.so.3.2.4
lalframe/utils/frtools.py
lalframe/utils/__init__.py

rename vendored libraries

as per here:
https://mail.python.org/pipermail/wheel-builders/2016-March/000082.html

Provide high-level library API

It would be great if this package would offer an API at the same level of the command line interface. My concrete use case is that I'd like to run auditwheel straight from setup.py as a build step after bdist_wheel within a custom command (the same could be done by subclassing build_ext). Currently I'm doing something like:

class bdist_wheel_and_repair(Command):
    user_options = []
    def initialize_options(self):
        pass
    def finalize_options(self):
        pass
    def run(self):
        self.run_command('bdist_wheel')
        if sys.platform.startswith('linux'):
            import auditwheel.main
            import unittest.mock
            wheel_path = next(item[2] for item in self.distribution.dist_files if item[0] == 'bdist_wheel')
            with unittest.mock.patch('sys.argv', [sys.argv[0], 'repair', wheel_path]):
                auditwheel.main.main()

How about something like auditwheel.repair(path) etc. to avoid mocking sys.argv?

auditwhell missing import print_function in python2

$pip show auditwheel
Name: auditwheel
Version: 1.5.0
Summary: Cross-distribution Linux wheels
Home-page: https://github.com/pypa/auditwheel
Author: Robert T. McGibbon
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python2.7/dist-packages
Requires: setuptools, wheel, typing, pyelftools

 $auditwheel
Traceback (most recent call last):
  File "/usr/local/bin/auditwheel", line 6, in <module>
    from auditwheel.main import main
  File "/usr/local/lib/python2.7/dist-packages/auditwheel/main.py", line 8, in <module>
    from . import main_addtag
  File "/usr/local/lib/python2.7/dist-packages/auditwheel/main_addtag.py", line 3, in <module>
    from .policy import (load_policies, get_policy_name, get_priority_by_name,
  File "/usr/local/lib/python2.7/dist-packages/auditwheel/policy/__init__.py", line 26
    print('Error: This tool only supports Linux', file=sys.stderr)
                                                      ^
SyntaxError: invalid syntax

$uname -a
Linux A10092495U 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Whitelist libresolv?

This library is part of glibc (similarly to libc, libm, librt, libpthread etc which are already whitelisted) and uses the same symbol versioning. I wonder if perhaps it is better not to bundle it?

Context: I ran into some trouble with the psycopg2 wheel, please see the details here: psycopg/psycopg2-wheels#2. If I replace the bundled version with the system version it works fine.

Missing library error from repaired wheel

I'm sorry not to have done a better job of debugging this but ..

I'm trying to build h5py wheels.

I think I correctly built and repaired the wheel, and it installs and tests OK on a Debian sid machine I have.

However, when I test the wheel on the docker image itself, I get:

python -c 'import h5py; h5py.run_tests()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/3.4.4/lib/python3.4/site-packages/h5py/__init__.py", line 24, in <module>
    from . import _errors
ImportError: libsz.so.2: cannot open shared object file: No such file or directory

Script to generate hdf5 and libsz libraries: https://gist.github.com/9279c5c368d61118cf1c
Script to build h5py wheel : https://gist.github.com/matthew-brett/0dbd3747cbd76ea8ca49
Unfixed wheel : http://nipy.bic.berkeley.edu/scipy_installers/tmp/unfixed/h5py-2.4.0-cp34-cp34m-linux_x86_64.whl
Fixed wheel : http://nipy.bic.berkeley.edu/scipy_installers/tmp/fixed/h5py-2.4.0-cp34-cp34m-linux_x86_64.whl
Script to run wheel test : https://gist.github.com/41eaad61fbf02072b2ba

libsz.so.2 is in the .libs directory of the h5py install, and rpath for the _errors extension does contain $ORIGIN/.libs.

Latest wheel==0.32.0 breaks auditwheel/wheeltools.py

Wheel 0.32.0 was released two days ago and has refactored its (private) API, breaking auditwheel:

Traceback (most recent call last):
  File "/usr/local/bin/auditwheel", line 11, in <module>
    sys.exit(main())
  File "/opt/_internal/cpython-3.6.6/lib/python3.6/site-packages/auditwheel/main.py", line 49, in main
    rval = args.func(args, p)
  File "/opt/_internal/cpython-3.6.6/lib/python3.6/site-packages/auditwheel/main_repair.py", line 43, in execute
    from .repair import repair_wheel
  File "/opt/_internal/cpython-3.6.6/lib/python3.6/site-packages/auditwheel/repair.py", line 13, in <module>
    from .wheeltools import InWheelCtx, add_platforms
  File "/opt/_internal/cpython-3.6.6/lib/python3.6/site-packages/auditwheel/wheeltools.py", line 14, in <module>
    from wheel.util import urlsafe_b64encode, open_for_csv, native  # type: ignore
ImportError: cannot import name 'open_for_csv'

https://travis-ci.org/python-pillow/pillow-wheels/jobs/435427307#L4860

The same thing hit multibuild and delocate: pypa/wheel#255.

Note that wheel officially has no public API:

It should be noted that wheel is not intended to be used as a library, and as such there is no stable, public API.

pypa/wheel#255

(We're working around the issue by pinning to the previous wheel==0.31.1.)

Support checking whole virtualenv path for dynamic linking closure?

It would be nice if auditwheel will support checking if path is closed against dynamic linking. What do you think? I know of https://github.com/matthew-brett/delocate but it's for MacOS and I'd like to have similar tool for Linux.

auditwheel should replace the linux tag with a manylinux1 tag instead of adding a new tag

auditwheel has a command named addtag to rename a wheel file to insert the manylinux1_x86_64 platform tag into its filename (in addition to the existing linux_x86_64) and similarly for manylinux1_i686 / linux_i686.

This is an issue because uploading wheel files with the linux_x86_64 / linux_i686 platform tags to warehouse or PyPI is forbidden (while manylinux1_x86_64/ manylinux1_i686 are authorized or will soon be).

Instead auditwheel repair should probably replace the linux based tags by manylinux1 tags whenever the repair operation succeeds.

Commit bit for @ehashman?

Hi all,

I'd like to propose @ehashman for a commit bit. See #43, #73 for some previous contributions, and she's interested in getting more involved with auditwheel maintenance.

Any objections? @rmcgibbo?

And mechanically, I think we would need someone with admin rights to the pypa organization to invite her to the manylinux team? @dstufft?

RecursionError in auditwheel show

Running auditwheel show on a wheel of https://github.com/anntzer/mpl_cairo compiled with gcc7.1 (which is definitely not manylinux1-compatible, I was just doing some tests) gives the following traceback:

Traceback (most recent call last):
  File "/home/antony/.local/bin/auditwheel", line 11, in <module>
    sys.exit(main())
  File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/main.py", line 49, in main
    rval = args.func(args, p)
  File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/main_show.py", line 28, in execute
    winfo = analyze_wheel_abi(args.WHEEL_FILE)
  File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/wheel_abi.py", line 73, in analyze_wheel_abi
    get_wheel_elfdata(wheel_fn)
  File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/wheel_abi.py", line 56, in get_wheel_elfdata
    ctx.path)
  File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/policy/external_references.py", line 53, in lddtree_external_references
    whitelist)  # type: List[str]
  File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/policy/external_references.py", line 39, in get_req_external
    whitelist) for lib in libs), libs)
  File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/policy/external_references.py", line 39, in <genexpr>
    whitelist) for lib in libs), libs)

<elided>

  File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/policy/external_references.py", line 39, in <genexpr>
    whitelist) for lib in libs), libs)
RecursionError: maximum recursion depth exceeded while calling a Python object

(Python 3.6.1 ArchLinux, auditwheel from pypi).

Question: how to point auditwheel to the correct library path

I receive the following error:

ValueError: Cannot repair wheel, because required library "libarb.so.2" could not be located

However, I know where this library is located. How can I point auditwheel to that location?

CXXABI symbol version in policy for manylinux is wrong

I believe there is a typo in the policy for the CXXABI symbol version. While the file lists "3.4.8" as the max permissible version, I dumped all the .gnu.version_d info from all the whitelisted libs inside the manylinux docker image and found that the max version was actually "1.3.1".

Even on a Debian jessie image (much more recent than the CentOS 5 manylinux image), I cannot find a CXXABI symbol with a version greater than 1.3.8.

Tests are throwing version exception for patchelf but not failing

The tests are full of the following traceback

Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.4.3/bin/auditwheel", line 10, in <module>
    sys.exit(main())
  File "/home/travis/virtualenv/python3.4.3/lib/python3.4/site-packages/auditwheel/main.py", line 49, in main
    rval = args.func(args, p)
  File "/home/travis/virtualenv/python3.4.3/lib/python3.4/site-packages/auditwheel/main_repair.py", line 81, in execute
    update_tags=args.UPDATE_TAGS)
  File "/home/travis/virtualenv/python3.4.3/lib/python3.4/site-packages/auditwheel/repair.py", line 81, in repair_wheel
    new_soname, new_path = copylib(src_path, dest_dir)
  File "/home/travis/virtualenv/python3.4.3/lib/python3.4/site-packages/auditwheel/repair.py", line 132, in copylib
    verify_patchelf()
  File "/home/travis/virtualenv/python3.4.3/lib/python3.4/functools.py", line 472, in wrapper
    result = user_function(*args, **kwds)
  File "/home/travis/virtualenv/python3.4.3/lib/python3.4/site-packages/auditwheel/repair.py", line 37, in verify_patchelf
    version)
ValueError: patchelf  found. auditwheel repair requirespatchelf >= 0.9.

Interestingly, the auditwheel tests are not failing, despite the tracebacks thrown.

The problem is that patchelf --version doesn't output anything on version 0.9. I've heard @geofft has submitted a patch to fix this upstream, and I've tested & confirmed the patch fixes the bug.

mechanism for "auditwheel repair" to add a local version identifier?

Hi,

Running auditwheel repair may generate subtly different versions of the wheel, depending on the exact version of so libraries being included.

When looking at the wheel or otherwise running pip freeze there is no externally visible difference though.

I was wondering if it would make sense to have auditwheel repair accept a new optional "local version identifier" [1] argument that would be appended to the repaired wheel version.

This would enable user to encode some information about the embedded dependencies in the repaired version.

[1] https://www.python.org/dev/peps/pep-0440/#local-version-identifiers

Do name mangling on individual symbols?

auditwheel uses patchelf to rename vendored libraries with unique names. This works wonderfully until something links against a library with the same symbol names and loads them before we get there. This scenario occurs when using uwsgi with cryptography's new manylinux1 wheels. Since uwsgi links against libssl/libcrypto itself we're seeing segfaults where it's calling between two different versions of the library (the one uwsgi loads globally and the one cryptography has vendored). You can see more discussion (and a better explanation from @njsmith) here: pyca/cryptography#3804 (comment)

One potential solution that was put forth would be to make it so patchelf is capable of name mangling individual symbols so that even in a scenario like the one above there's no issue with namespace collision. This sounds very challenging but as more and more manylinux1 wheels start being shipped it may be the right long term solution.

Check UCS2 vs UCS4

For Py < 3.2, the manylinux1 policy requires building against UCS4 Python, so we have to check for this in extensions modules by looking through the symbol table for the use of any PyUnicodeUCS2_* functions, and give a helpful error message.

bugs with cffi

I found some bugs trying to build and repair cffi. fixes in the works.

Inconsistency detected by ld.so: dl-version.c: 205: _dl_check_map_versions: Assertion `needed != NULL' failed!

Update for the weary traveler: You may be using a too old version of patchelf, try compiling it from source and using that instead.

I'm getting this error when I try to import the module after install my repaired wheel in a fresh VM (or trying to import it directly from the .so on the machine where it was built).

Inconsistency detected by ld.so: dl-version.c: 205: _dl_check_map_versions: Assertion `needed != NULL' failed!

The wheel is build for this package (https://github.com/safijari/pyOpenKarto/blob/python-devel/setup.py) in python2.7. I run auditwheel and get:

(auditwheel_pipenv-zl3561H8) > $ auditwheel repair py_openkarto-0.0.1-cp27-cp27mu-linux_x86_64.whl --plat linux_x86_64
Repairing py_openkarto-0.0.1-cp27-cp27mu-linux_x86_64.whl
Grafting: /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 -> .libsopenkarto/libstdc++-637c843a.so.6.0.25
Grafting: /usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1 -> .libsopenkarto/libboost_system-ca31776d.so.1.65.1
Grafting: /lib/x86_64-linux-gnu/libm-2.27.so -> .libsopenkarto/libm-2-24310d2a.27.so
Grafting: /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1 -> .libsopenkarto/libboost_thread-4e4bd896.so.1.65.1
Grafting: /lib/x86_64-linux-gnu/libc-2.27.so -> .libsopenkarto/libc-2-cd7c1a03.27.so
warning: working around a Linux kernel bug by creating a hole of 2101248 bytes in ‘.libsopenkarto/libc-2-cd7c1a03.27.so’
Grafting: /lib/x86_64-linux-gnu/libpthread-2.27.so -> .libsopenkarto/libpthread-2-1032040b.27.so
warning: working around a Linux kernel bug by creating a hole of 2076672 bytes in ‘.libsopenkarto/libpthread-2-1032040b.27.so’
Grafting: /lib/x86_64-linux-gnu/libgcc_s.so.1 -> .libsopenkarto/libgcc_s-b10d5179.so.1
Grafting: /lib/x86_64-linux-gnu/librt-2.27.so -> .libsopenkarto/librt-2-110b6497.27.so
Setting RPATH: openkarto.so to "$ORIGIN/.libsopenkarto"
Previous filename tags: linux_x86_64
No filename tags change needed.
Previous WHEEL info tags: cp27-cp27mu-linux_x86_64
No WHEEL info change needed.

Fixed-up wheel written to /home/jari/Pipenvs/karto/pyOpenKarto/dist/wheelhouse/py_openkarto-0.0.1-cp27-cp27mu-linux_x86_64.whl

Nothing seems wrong with that output as far as I can tell. Pls halp :(

Update: Hmm, I moved all the .so files elsewhere and moved them back one by one, once I moved back this file...

Grafting: /usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1 -> .libsopenkarto/libboost_system-ca31776d.so.1.65.1

that's when I get the crash... hmm. Unsure if this is to blame as I have another package like this which doesn't use boost and gives the same crash after auditwheel.

modules statically linked to libpythonx.y.a are show as compliant

Please consider checking that a module library has undefined python symbols to verify that it has not been statically linked against libpython.a

I am using CMake as a build tool so it's a bit more manual than distutils for linking. I had previously been linking my module to libpython as a shared library. In testing with the recent manylinux docker images, I created a 32-bit wheel what was linked against it's static libpyton. When I ran audit wheel is said it was compliant.

I think that if the module is statically linked it should raise a flag with auditwheel.

Repair does not handle recursive libraries correctly

Using the latest manylinux image (and so hopefully the latest auditwheel version) I have the problem that repair does not correctly relink all libraries. The dependency hierarchy is as follows:

parquet.so is a Python module (Cython generated) that depends on the C++ libraries libpyarrow.so and libarrow.so directly.
libpyarrow.so is a pure C++ helper library but also depends on libarrow.so directly.

parquet.so has the correct links after the repair but libpyarrow.so does not.

Both libs parquet.so and libpyarrow.so are part of the wheel, libarrow.so is in this case an external one).

Console output:

Repairing pyarrow-0.1.0.dev0-cp27-cp27mu-linux_x86_64.whl
Grafting: /usr/lib/libarrow.so -> pyarrow/./libarrow-17b4bbba.so
Grafting: /usr/lib/libboost_filesystem.so.1.60.0 -> pyarrow/./libboost_filesystem-d4710d70.so.1.60.0
Grafting: /usr/lib/libarrow_parquet.so -> pyarrow/./libarrow_parquet-745cd113.so
Grafting: /usr/lib/libparquet.so -> pyarrow/./libparquet-3ec6b6fd.so
Grafting: /usr/lib/libboost_system.so.1.60.0 -> pyarrow/./libboost_system-17a2ee5a.so.1.60.0
Grafting: /usr/lib/libarrow_io.so -> pyarrow/./libarrow_io-f94b9eae.so
Setting RPATH: pyarrow/schema.so to "$ORIGIN/."
Setting RPATH: pyarrow/scalar.so to "$ORIGIN/."
Setting RPATH: pyarrow/config.so to "$ORIGIN/."
Setting RPATH: pyarrow/table.so to "$ORIGIN/."
Setting RPATH: pyarrow/array.so to "$ORIGIN/."
Setting RPATH: pyarrow/error.so to "$ORIGIN/."
Setting RPATH: pyarrow/io.so to "$ORIGIN/."
Setting RPATH: pyarrow/parquet.so to "$ORIGIN/."
Previous filename tags: linux_x86_64
New filename tags: manylinux1_x86_64
Previous WHEEL info tags: cp27-cp27mu-linux_x86_64
New WHEEL info tags: cp27-cp27mu-manylinux1_x86_64

Library linkages:

% readelf --dynamic pyarrow/parquet.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libpyarrow.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow-17b4bbba.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_io-f94b9eae.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_parquet-745cd113.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

% readelf --dynamic pyarrow/libpyarrow.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libarrow.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_io.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_parquet.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

Happy to fix this on my own but I would need some pointers where I should start to investigate!

How to handle C++?

First of all I think this initiative is amazing and really hope this will make portable Linux binaries trivial.

I've had some experience with this on the company I work for. Basically we came up with the same solution, except that we compile under a CentOS 5 chroot.

But one of the problems we faced was libstdc++. We also use gcc 4.8.5, which means the libstdc++ version is 3.4.19.

CentOS default libstdc++ is 3.4.12, therefore if you are compiling C++ code using this approach, you might have to run the executable only on CentOS 7 and earlier.

So the most obvious solution would be to just ship libstdc++. This is what we did, but then we got missing symbols on recent Linux distributions because they have a libstdc++ newer than 3.4.19, and some crucial system libraries (specially OpenGL implementations) were linked against it.

Not the most beautiful solution, but we ended up having to create an initialization script that would check which version is newer (the shipped or system one), and set LD_LIBRARY_PATH to the shipped one if necessary.

I know that most of base libraries are written in C, but even though I think this is an important issue, specially if someone is using OpenGL (I know that at least Mesa and Nvidia proprietary implementation use C++).

So have anyone else thought about this problem?

Compatibility with Python 2.7 (feature request)

Is there a specific reason why auditwheel does not support Python 2.7 (besides it requiring more work to make the code Python2/3 portable)?

I got a simple error from a print call in main_addtag.py, line 49.

policy.json: Add libXt.so.6 to whilelist

Working on creating wheels for VTK, I found out that libXt.so.6 wasn't specified in policy.json and was copied in .libs directory.

Would it make sense to add it ?

What could I do to check that is it reasonable to add this library to the whitelist ? Is it mater of having a consistent API/API across all versions of linux we support ?

See https://en.wikipedia.org/wiki/X_Toolkit_Intrinsics

Cc: @matthew-brett @thewtex

check for binaries in purelib

To help in situations like #28, it would be nice if auditwheel could check for impure files in purelib and complain if they're found.

auditwheel incorrectly does not determine if shared object is python extension with pyelftools 0.24

Because since that version symbol and section names are strings in elftools and it mentioned in CHANGES.

Here is commit changed behaviour - eliben/pyelftools@108eaea

Auditwheel should check for references to PyFPE_jbuf, and error out if present

See:
python/peps#166
numpy/numpy#8415 (comment)

Namespace packages cause unexpected errors in `repair`

In a project that auditwheel show reports is manylinux1 compatible and that auditwheel repair successfully repairs, adding the namespace_packages argument to the setuptools.setup call results in auditwheel show still reporting that the project is manylinux1 compatible, however, auditwheel repair breaks:

Traceback (most recent call last):
  File "/usr/local/bin/auditwheel", line 11, in <module>
    sys.exit(main())
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main.py", line 49, in main
    rval = args.func(args, p)
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main_repair.py", line 81, in execute
    update_tags=args.UPDATE_TAGS)
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/repair.py", line 68, in repair_wheel
    raise RuntimeError('Is this wheel malformatted? Or a bug?')
RuntimeError: Is this wheel malformatted? Or a bug?

This is with auditwheel 1.3.0 within the manylinux1 x64 docker image.

Here is a minimal testcase (n.b. requires cython).

Auditwheel 1.6 breaks numpy / matplotlib wheel builds

Auditwheel 1.6 seems to be failing to detect and graft the dependencies of the wheels.

For new failure example see:

https://api.travis-ci.org/jobs/227872764/log.txt?deansi=true

For failure fixed by downgrading auditwheel to 1.5:

https://api.travis-ci.org/jobs/227948047/log.txt?deansi=true

With the downgrade, I see this in the log (second link, just above):

Grafting: /usr/local/lib/libopenblasp-r0.2.18.so -> numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
Grafting: /usr/lib64/libgfortran.so.3.0.0 -> numpy/.libs/libgfortran-ed201abd.so.3.0.0
Setting RPATH: numpy/core/multiarray.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/linalg/_umath_linalg.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/linalg/lapack_lite.so to "$ORIGIN/../.libs"

These are missing in the run with new auditwheel 1.6 (first link above).

Seems to be the same problem for matplotlib:

https://api.travis-ci.org/jobs/227868822/log.txt?deansi=true

Here it looks like the libpng library was not grafted into the matplotlib wheel.

Option to strip binaries generated by auditwheel

Auditwheel is a great tool and is used by a lot of common libraries to build their wheel packages. I'm most interested in numpy and they mentioned that they use this library to package their wheels. Unfortunately somewhere in their build process the binaries are modified by patchelf such that strip doesn't work.

Is it possible to add the option to strip binaries directly within the auditwheel process?

Stripped binaries are critical for deployment, particularly of lambda functions and I'm unable to do the strip myself once installed.

A lot of people would benefit from this change in terms of saved space in deployment.

Feature Request: Configurable policy document for self-hosted PyPI repositories

Is it possible to have a configurable policy document, perhaps a command line option to specify a different policy document that would bundle only the libraries specified in this user-specified policy doc.

Use case:

For organizations with self-hosted PyPI repositories, the process for bundling shared libraries can conform to a standard set by the organization itself. For instance, for orgs that run Alpine Linux, auditwheel repair can omit shared libraries common to Alpine Linux, e.g. musl c.

By allowing a user to specify their own policy for determining what to dependencies bundle they can reduce wheel sizes.

NameError: name 'libname' is not defined

I applied auditwheel to a project named E-Cell.
But there was NameError for this.
If you have any suggestions in this regard, please let me know.

[root@db4ffc7bd536 python]# auditwheel show dist/ecell-4.0.0-cp27-cp27mu-linux_x86_64.whl 

ecell-4.0.0-cp27-cp27mu-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references the following external versioned symbols in
system-provided shared libraries: CXXABI_1.3.1, GLIBC_2.4,
GLIBCXX_3.4, GCC_3.0.

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib64/libc-2.5.so",
    "libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
    "libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
    "libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0",
    "libhdf5.so.9": null,
    "libhdf5_cpp.so.9": null,
    "libm.so.6": "/lib64/libm-2.5.so",
    "libpthread.so.0": "/lib64/libpthread-2.5.so",
    "libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8"
}

In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:

libgsl.so.0, libgslcblas.so.0, libhdf5.so.9, libhdf5_cpp.so.9
[root@db4ffc7bd536 python]# auditwheel repair dist/ecell-4.0.0-cp27-cp27mu-linux_x86_64.whl 
Repairing ecell-4.0.0-cp27-cp27mu-linux_x86_64.whl
Traceback (most recent call last):
  File "/usr/local/bin/auditwheel", line 11, in <module>
    sys.exit(main())
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main.py", line 49, in main
    rval = args.func(args, p)
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main_repair.py", line 81, in execute
    update_tags=args.UPDATE_TAGS)
  File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/repair.py", line 79, in repair_wheel
    libname)
NameError: name 'libname' is not defined

Vendored library cannot be loaded

I used the manylinux docker infrastructure to build:

http://travis-wheels.scikit-image.org/netCDF4-1.2.3.1-cp34-cp34m-manylinux1_x86_64.whl

Upon import, I see:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/stefan/envs/py3/lib/python3.4/site-packages/netCDF4/__init__.py", line 3, in <module>
    from ._netCDF4 import *
ImportError: libhdf5_hl-9c7ba457.so.10.0.2: cannot open shared object file: No such file or directory

Looking at the vendored libraries, the RPATH is invalid for libnetcdf (not for libhdf5)--possibly has something to do with original libnetcdf library already having an RPATH entry.

Build recipe can be found here:

https://github.com/stefanv/manylinux-builds/blob/build_netcdf/build_netcdfs.sh

It was run on yesterday's version of the manylinux docker image.

/cc @njsmith @matthew-brett

auditwheel show also works on conda packages

Not a bug report, rather a note on a nice feature. I found that auditwheel can also be used to audit a conda package for external shared libraries. I have not tested if any of the other sub-commands work. Thanks for this tool, it is really helpful.

$ auditwheel show linux-64-tensorflow-0.7.1-py35_1.tar.bz2

linux-64-tensorflow-0.7.1-py35_1.tar.bz2 is consistent with the
following platform tag: "linux_x86_64".

The wheel references the following external versioned symbols in
system-provided shared libraries: GLIBC_2.14, CXXABI_1.3.5, GCC_3.0,
GLIBCXX_3.4.19.

This constrains the platform tag to "linux_x86_64". In order to
achieve a more compatible tag, you would to recompile a new wheel from
source on a system with earlier versions of these libraries, such as
CentOS 5.

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib64/libc-2.5.so",
    "libdl.so.2": "/lib64/libdl-2.5.so",
    "libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
    "libm.so.6": "/lib64/libm-2.5.so",
    "libpthread.so.0": "/lib64/libpthread-2.5.so",
    "libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8",
    "libz.so.1": "/lib64/libz.so.1.2.3"
}

In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:

libz.so.1

Whitelisting of libraries shipped by other wheels

In turbodbc we link against pyarrow in such a fashion that turbodbc picks up the Arrow libraries via a relative RPATH in the same virtualenv. Thus we don't need to include them in the manylinux1 wheel of Turbodbc. With auditwheel's current behaviour they are sadly grafted into turbodbc's wheel. To fix this, two options come to my mind:

whilelist all libraries that are contained in the virtualenv
explicitly whitelist some libraries via a commandline option

I'm happy to implement one of these approaches but I would like to hear what the preferred one would be.

does `auditwheel` check whether referenced versions of `glibc` comply to PEP 513?

Using auditwheel on a wheel that:

contains a compiled Cython extension
is linked dynamically to CPython
contains a statically linked third-party library
was built on a Debian Stretch (Linux dhcp-7-187 4.9.0-1-amd64 #1 SMP Debian 4.9.6-3 (2017-01-28) x86_64 GNU/Linux)

the output is:

auditwheel show dist/dd-0.5.2.dev0+f9ea13bad6ef42cccbe0df9dafd66bea265a490d-cp35-cp35m-linux_x86_64.whl 

dd-0.5.2.dev0+f9ea13bad6ef42cccbe0df9dafd66bea265a490d-cp35-cp35m-linu
x_x86_64.whl is consistent with the following platform tag:
"manylinux1_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libc.so.6 with versions {'GLIBC_2.14',
'GLIBC_2.7', 'GLIBC_2.2.5'}

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib/x86_64-linux-gnu/libc-2.24.so",
    "libpthread.so.0": "/lib/x86_64-linux-gnu/libpthread-2.24.so"
}

I am confused by the fact that GLIBC 2.14 and 2.7 are listed, but PEP 513 requires that GLIBC <= 2.5 (which agrees with policy.json). Looking closer at the extension module:

ldd -r -v build/lib.linux-x86_64-3.5/dd/cudd.cpython-35m-x86_64-linux-gnu.so 
	linux-vdso.so.1 (0x00007ffe1328e000)
	libpython3.5m.so.1.0 => /some/path/python35/lib/libpython3.5m.so.1.0 (0x00007f3d50ed5000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3d50c75000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3d508d7000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3d506d3000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f3d504d0000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3d501ca000)
	/lib64/ld-linux-x86-64.so.2 (0x00005580a0e8a000)

	Version information:
	build/lib.linux-x86_64-3.5/dd/cudd.cpython-35m-x86_64-linux-gnu.so:
		libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
	/some/path/python35/lib/libpython3.5m.so.1.0:
		libdl.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libdl.so.2
		libutil.so.1 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libutil.so.1
		libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
		libpthread.so.0 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libpthread.so.0
		libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
		libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.9) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.17) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.6) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
	/lib/x86_64-linux-gnu/libpthread.so.0:
		ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
		ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
		libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
	/lib/x86_64-linux-gnu/libc.so.6:
		ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
		ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
	/lib/x86_64-linux-gnu/libdl.so.2:
		ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
		libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
	/lib/x86_64-linux-gnu/libutil.so.1:
		libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
	/lib/x86_64-linux-gnu/libm.so.6:
		ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
		libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6

The various GLIBC dependencies are on /lib/x86_64-linux-gnu/libc.so.6. Running the library says:

/lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Debian GLIBC 2.24-9) stable release version 2.24, by Roland McGrath et al.
Copyright (C) 2016 Free Software Foundation, Inc.
...

So, does auditwheel not check for the referenced versions?

The source code suggests that the version sets are checked, but there is also a whitelist that seems to be processed in parallel with versioned symbols, inside wheel_abi.

ncurses 5 -> 6 transition means we probably need to drop some libraries from the manylinux whitelist

The PEP 513 whitelist includes libpanelw.so.5 and libncursesw.so.5, both of which are part of ncurses version 5.

But apparently the new hostness is ncurses 6, which breaks ABI with ncurses 5. I'm told that Fedora 24 and successors have already switched to ncurses 6, with the ncurses 5 libraries being available in a compatibility package, but not installed by default. So... I guess these have to be dropped from the whitelist.