build: Setting path to system libs about tensorstore HOT 4 OPEN

Follow-up question: If I only need support for precomputed and not for n5 or zarr, is there a way to disable n5 support to simplify the build? (Currently, the segfault I'm encoutering in clang happens to occur while compiling blosc-related stuff.)

from tensorstore.

There isn't a particularly nice way to configure which drivers are included, but you can edit python/tensorstore/BUILD to exclude drivers you don't want from the _tensorstore rule. It would be possible to add a flag to control that, though, without needing to modify the BUILD file.

Regarding mac (or windows) binary wheel: we do plan to generate them, we just haven't set up the necessary infrastructure to do it yet, unfortunately. What version of clang/xcode are you using? Is there an error associated with the segfault?

Regarding specifying a path to system libraries: I think you can accomplish that by setting the following environment variables when building:

CPLUS_INCLUDE_PATH=/path/to/conda/include/dir
BAZEL_LINKOPTS=-Wl,-rpath=/path/to/conda/lib/dir:-L/path/to/conda/lib/dir

I'm not familiar with how conda packages are supposed to be built, so I'm not sure whether you are supposed to set rpath or not.

The CPLUS_INCLUDE_PATH unfortunately only applies to C++ sources, and there doesn't seem to be a corresponding bazel environment variable for C sources, which should mostly be fine, since all of the tensorstore code is C++, but that may be a problem if you use a mix of system and bundled libraries, since the bundled libraries do include C code. You can alternatively specify --copt=-I/path/to/conda/include/dir when invoking bazel (i.e. bazelisk.py) directly, or modify .bazelrc.

from tensorstore.

Thanks for the tips. I'm still running into issues, but I don't expect you to resolve them for me right now. I can put this aside for a while.

For what it's worth, here's what I've tried so far:

First, I made a few minor fixes to some bazel scripts. See my branch here. Then I tried the following:

conda create -n ts -y bazel bzip2 blosc=1.16.3 curl jpeg lz4 nasm libnghttp2 snappy xz zlib zstd python=3.7 numpy
source activate ts
export TENSORSTORE_SYSTEM_LIBS=org_sourceware_bzip2,org_blosc_cblosc,se_haxx_curl,jpeg,org_lz4,nasm,org_nghttp2,com_google_snappy,org_tukaani_xz,net_zlib,com_facebook_zstd
export CPLUS_INCLUDE_PATH=${CONDA_PREFIX}/include
export BAZEL_LINKOPTS=-Wl,-rpath=${CONDA_PREFIX}/lib:-L${CONDA_PREFIX}/lib
bazel build --action_env=CPLUS_INCLUDE_PATH --sandbox_debug -c opt //python/tensorstore:_tensorstore__shared_objects

which yields the following output:

INFO: Analyzed target //python/tensorstore:_tensorstore__shared_objects (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /Users/bergs/workspace/tensorstore/tensorstore/driver/n5/BUILD:9:23: undeclared inclusion(s) in rule '//tensorstore/driver/n5:blosc_compressor':
this rule is missing dependency declarations for the following files included by 'tensorstore/driver/n5/blosc_compressor.cc':
  '/opt/miniconda/envs/ts/include/blosc.h'
  '/opt/miniconda/envs/ts/include/blosc-export.h'
Target //python/tensorstore:_tensorstore__shared_objects failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6.255s, Critical Path: 6.06s
INFO: 10 processes: 10 darwin-sandbox.
FAILED: Build did NOT complete successfully

So, it seems bazel is not happy about using headers that reside outside of its own build tree. (The exact error is not always the same, since the build is multi-threaded. Sometimes it complains about a different header, e.g. curl.h or lzma.h.)

I have no knowledge of bazel, so I don't know if this can be fixed. I did notice a few command-line options that looked promising:

--nostrict_system_includes
--noincompatible_validate_top_level_header_inclusions

...but they seem to make no difference, or I'm using them incorrectly.

Anyway, even if I could get past that issue, I might still encounter a segfault in clang anyway, so maybe this approach won't buy me anything.

Furthermore, to have any hope of contributing a recipe to conda-forge, I would have to figure out how to make bazel use the version of clang that conda-forge ships, rather than Xcode. That involves configuring a bazel custom toolchain. Apparently the conda-forge folks figured out how to do this for their tensorflow recipe, but unfortunately that recipe builds a fairly old version of tensorflow, and uses a rather old version of bazel (0.24). When I tried to use that toolchain with a newer version of bazel, I ran into errors. Maybe someday I'll try to figure out how to upgrade it for bazel 3.

One more question, in case I come back to this at some point: For jpeg support, do you rely on any jpeg-turbo features, or would plain libjpeg suffice?

from tensorstore.

As far as jpeg support, libjpeg should suffice.

It looks like bazel has special treatment for system headers. It seems like additional system headers might have to be specified via a custom toolchain as you noted, but unfortunately I don't know how to set that up. I will see if someone on the Bazel team can offer some help.

from tensorstore.