Build issues in a clean environment about tensorstore HOT 12 CLOSED

Per the installation instructions, you need GCC 9 or Clang 8.

On CentOS you can use the Redhat devtoolset-9 package:

RUN yum -y update && yum -y install \
    devtoolset-9-gcc-c++ \
    && yum clean all

ENV PATH="/opt/rh/devtoolset-9/root/usr/bin:$PATH"

I'm working on setting up some automation for building binary wheels using the manylinux2014 container as a base. The following Dockerfile works for building wheels, though I haven't set up the automation yet:

ARG BASE=quay.io/pypa/manylinux2014_x86_64

FROM $BASE

LABEL description="A docker image for building portable Python linux binary wheels using modern GCC"
LABEL maintainer="[email protected]"

RUN yum -y update && yum -y install \
    devtoolset-9-gcc-c++ \
    && yum clean all

ENV PATH="/opt/rh/devtoolset-9/root/usr/bin:$PATH"

manylinux1 and manylinux2010 are more challenging since the pre-built Bazel binaries don't work (and also it is necessary to build a recent GCC from source, as the devtoolset-9 package isn't available).

Note: in the manylinux2014 container, the Python versions are in /opt/python/.

from tensorstore.

Hi Jeremy,

I tried both Stuart's CENTOS image and the one you specified above, and I cannot get tensorstore to install.
Maybe I am doing something slightly wrong. Rather than debug what I am doing, I am happy to use whatever image works for you. If you have a DockerFile that builds tensorstore, can you attach it? Thanks!!

from tensorstore.

Per the installation instructions, you need GCC 9 or Clang 8.

Ah, I missed that. Thanks!

I tried both Stuart's CENTOS image and the one you specified above, and I cannot get tensorstore to install.

The following Dockerfile works for me. (Add python3 and python3-devel)

ARG BASE=quay.io/pypa/manylinux2014_x86_64

FROM $BASE

LABEL description="A docker image for building portable Python linux binary wheels using modern GCC"
LABEL maintainer="[email protected]"

RUN yum -y update && yum -y install \
    python3 \
    python3-devel \
    devtoolset-9-gcc-c++ \
    && yum clean all

ENV PATH="/opt/rh/devtoolset-9/root/usr/bin:$PATH"

RUN python3 -m pip install tensorstore -vv

CMD [ "/bin/bash" ]

from tensorstore.

When I build this container, I get a runtime error. A symbol cannot be found. Is there another path that should be set?

import tensorstore as ts

File "", line 1, in
File "/usr/local/lib64/python3.6/site-packages/tensorstore/init.py", line 18, in
from ._tensorstore import *
ImportError: /usr/local/lib64/python3.6/site-packages/tensorstore/_tensorstore.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZTINSt6thread6_StateE

from tensorstore.

The easiest thing to try is probably just to start with a base image for Ubuntu 19.10, which uses gcc-9 by default.

from tensorstore.

I can reproduce this, and I'm looking into this.

Building a wheel inside the container then using it outside the container works on my system, but my system libstdc++.so.6 contains that symbol, while it is missing in /lib64/libstdc++.so.6 inside the container.

I think it might have something to do with the unusual way in which the devtoolset-9 package was created in order to provide a newer compiler on top of an older base system.

I think stuart's suggestion is a good one --- if you just want to use tensorstore inside a container, using a more recent base image which has a gcc-9 package available will almost certainly work.

Still, I'll try to get to the bottom of this since I do want tensorstore packages to be buildable for older systems.

from tensorstore.

In the meantime, this container builds successfully:

FROM ubuntu:19.10

RUN apt-get update \
 && apt-get install -y python3-dev \
                       python3-pip \
                       build-essential

RUN python3 -m pip install tensorstore -vv

RUN python3 -c 'import tensorstore'

CMD [ "/bin/bash" ]

from tensorstore.

This appears to be due to bazelbuild/bazel#10327

While that bug is listed as closed, the issue has not actually been fixed in the most recent bazel release, 3.0.0.

In current bazel master (but unfortunately not yet in any released version), there is a flag --incompatible_linkopts_to_linklibs (bazelbuild/bazel#10905)

https://docs.bazel.build/versions/master/command-line-reference.html#flag--incompatible_linkopts_to_linklibs

Once that flag is available in a released version of bazel, I'll go ahead and enable it which should fix the problem.

In the meantime, the following workaround works:

BAZEL_LINKLIBS=-lstdc++ python3 -m pip install tensorstore -vv

I'll see about integrating this workaround into the build scripts so that it is not needed to be done by users.

from tensorstore.

As a follow-up to this, I tried using a special gcc9 container (the newest versions of ubuntu give me build problems for other things). The following fails (though it gets farther than my attempts before using gcc9) -- error message is quite cryptic and long.

FROM gcc:9.3

RUN apt-get update
RUN apt-get install -y python3-dev
python3-pip
build-essential

RUN python3 -m pip install numpy
RUN python3 -m pip install pillow
RUN python3 -m pip install gunicorn
RUN python3 -m pip install google-cloud-storage
RUN python3 -m pip install flask
RUN python3 -m pip install flask-cors
RUN python3 -m pip install scipy

RUN python3 -m pip install tensorstore -vv
CMD bash

from tensorstore.

I can't reproduce the problem locally. I copied that into a Dockerfile, ran docker build, which completed successfully, then ran the container and was able to import tensorstore.

from tensorstore.

However, we hope to have binary package generation set up soon, which will simplify the installation.

from tensorstore.

That's really strange but I went to rebuild from scratch and it works now. Sorry for that. On the positive side, it is another docker container people can use if they couldn't use the latest red hat or ubuntu.

from tensorstore.