ali5h / rules_pip Goto Github PK

pip package rules for bazel that are fast (incremental fetch), support different python versions and work with all kinds of packages (i.e. packages with namespaces)

License: MIT License

Python 60.25% Shell 2.50% Starlark 37.24%

bazel python pip

rules_pip's Introduction

Bazel pip rules

Overview

This repository provides support for installing pip packages. Usage is similar to rules_python.

Main features

Fetch pip packages incrementally. Fetch happens only if needed by a target.
Support for namespace packages
Reproducible wheel builds.
Support for passing arguments to pip_install. You can optionally pass the same arguments that pip install accepts to pip_install. For example, if you have wheels for all the packages you can have pip_install(["--only-binary", ":all"]).
Support for python version. The assumption is that each python version has its own requirements file and the python binary of that version exists in the global environment.

requirements file

For full hermetic builds always use a requirements files that is produced by pip-compile. Also pre-build the wheels in advance. You can run pip-compile as:

$ pip-compile --allow-unsafe requirements.in

--allow-unsafe is important. For example some packages need setuptools and we need it to be in requirements file.

Otherwise, you can set compile=True in pip_import rule and the rule will try to compile the requirements file. But this process is fragile.

Setup

Add the following to your WORKSPACE file:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "com_github_ali5h_rules_pip",
    strip_prefix = "rules_pip-<revision>",
    sha256 = "<revision_hash>",
    urls = ["https://github.com/ali5h/rules_pip/archive/<revision>.tar.gz"],
)

load("@com_github_ali5h_rules_pip//:defs.bzl", "pip_import")

pip_import(
   name = "pip_deps",
   requirements = "//path/to:requirements.txt",

   # default value is "python"
   # python_interpreter="python3",

   # or specify a python runtime label
   # python_runtime="@python3_x86_64//:bin/python3",

   # set compile to true only if requirements files is not already compiled
   # compile = True
)

load("@pip_deps//:requirements.bzl", "pip_install")
pip_install([<optional pip install args])

Target dependencies

To use pip packages you can

add dependencies via requirement macro as

load("@pip_deps//:requirements.bzl", "requirement")

py_binary(
    name = "main",
    srcs = ["main.py"],
    deps = [
        requirement("pip-module")
    ]
)

use package aliases as

py_binary(
    name = "main",
    srcs = ["main.py"],
    deps = [
        "@pip_deps//:pip-module"
    ]
)

rules_pip's People

Contributors

Stargazers

Watchers

rules_pip's Issues

run pip install as part of ci

we must keep third_party/py and src/requirements.txt in sync.

Issue installing opencv-python

So one of my coworkers recently started using the opencv dependency. This project has a very large C component but thankfully they provide a number of binary wheels for different platforms.

For local development on macOS pip_install() has been able to install the correct wheel opencv_python-4.5.1.48-cp38-cp38-macosx_10_13_x86_64.whl. However, during our CI process pip_install() is instead downloading the source and compiling which can take over 15 minutes.

If I manually pip install on one of our CI instances (pip3 install opencv-python) it finds and downloads opencv_python-4.5.1.48-cp38-cp38-manylinux2014_x86_64.whl.

Any thoughts as to what might be going on here or how to debug?

add support for custom rules for a package

currently we have special rules for numpy, be able to add custom rules to BUILD file of a specific package would be nice

platform flag does not appear to compile all resources correctly

Hello,

I'm attempting to use the grpc library that depends on a built-in cython package. This appears to fail to build even when pip_install is passed the platform flag (pip_install(["--platform=linux_x86_64"]).

  File "/app/qa_configurator/qa_configurator.binary.runfiles/pypi__38__google_api_core_1_16_0/google/api_core/gapic_v1/config.py", line 23, in <module>
    import grpc
  File "/app/qa_configurator/qa_configurator.binary.runfiles/pypi__38__grpcio_1_23_0/grpc/__init__.py", line 23, in <module>
    from grpc._cython import cygrpc as _cygrpc
ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/app/qa_configurator/qa_configurator.binary.runfiles/pypi__38__grpcio_1_23_0/grpc/_cython/__init__.py)

I'm not sure if this is due to grpc using an internal dependency or not. I'm curious if there are any workarounds to this. Thanks!

`pip_install(["--platform linux_x86_64"]` running `pip wheel` and result in `no such option: --platform linux_x86_64`

I have the identical issue as explained here: bazelbuild/rules_python#260 (comment) and I'm trying to implement the suggestion from here: bazelbuild/rules_python#260 (comment).

My WORKSPACE:

## PIP install libs for python
http_archive(
    name = "com_github_ali5h_rules_pip",
    strip_prefix = "rules_pip-3.0.0",
    sha256 = "630a7cab43a87927353efca116d20201df88fb443962bf01c7383245c7f3a623",
    urls = ["https://github.com/ali5h/rules_pip/archive/3.0.0.tar.gz"],
)

load("@com_github_ali5h_rules_pip//:defs.bzl", "pip_import")

pip_import(
    name = "pip_deps",
    requirements = "//third_party/python_requirements:requirements.txt",
    python_interpreter="python3",
)

load("@pip_deps//:requirements.bzl", "pip_install")
pip_install(["--platform linux_x86_64", "--only-binary", ":all"])

Unfortunately, I get the following error:

INFO: Repository pypi__39__grpcio_1_37_1 instantiated at:
  /.../WORKSPACE:98:12: in <toplevel>
  /private/var/tmp/.../e3523c944af0bdd86cccbc59a8209df1/external/pip_deps/requirements.bzl:85:16: in pip_install
Repository rule whl_library defined at:
  /private/var/tmp/.../e3523c944af0bdd86cccbc59a8209df1/external/com_github_ali5h_rules_pip/defs.bzl:126:30: in <toplevel>
ERROR: An error occurred during the fetch of repository 'pypi__39__grpcio_1_37_1':
   Traceback (most recent call last):
	File "/private/var/tmp/.../e3523c944af0bdd86cccbc59a8209df1/external/com_github_ali5h_rules_pip/defs.bzl", line 124, column 13, in _whl_impl
		fail("whl_library failed: %s (%s)" % (result.stdout, result.stderr))
Error in fail: whl_library failed:  (
Usage:
  whl.py install [options] <requirement specifier> [package-index-options] ...
  whl.py install [options] -r <requirements file> [package-index-options] ...
  whl.py install [options] [-e] <vcs project url> ...
  whl.py install [options] [-e] <local project path> ...
  whl.py install [options] <archive url/path> ...

no such option: --platform linux_x86_64
)

I'm confused why the argument is being passed (as it seems) to pip wheel and not only pip install.

How should I solve this issue?

support '--extra-index-url'?

does rules_pip support extra index url?
I tried

pip_install(["--extra-index-url", "<url>"])

and

pip_install(["--extra-index-url=<url>"])

and also adding "--extra-index-url " into requirements.in

but none of above work.

Configurable download of cross platform extensions

Hi,

I've begun working with rules_docker and creating py3 images. However, we have a dependency on grpcio which of course has a native extension built in. Since the python binary is built on the host machine it's bundling the Darwin version of the extension into the linux container.

I've had success passing ["--platform=manylinux2010_x86_64"] to pip_install but was wondering if this was in anyway configurable.

I stumbled across this issue bazelbuild/rules_python#260 where you mentioned that it was possible to define multiple repos and use select to switch between them.

I took a first pass at this using the alias feature that was recently added to rules_pip:

WORKSPACE

# Linux 
pip_import(
    name = "py3_deps_manylinux2010_x86_64",
    compile = False,
    python_interpreter = "python3",
    requirements = "//3rdparty/python3:requirements.transitive.txt",
)

load(
    "@py3_deps_manylinux2010_x86_64//:requirements.bzl",
    pip3_manylinux2010_x86_64_install = "pip_install",
)

pip3_manylinux2010_x86_64_install(["--platform=manylinux2010_x86_64", "--prefer-binary"])

# Darwin
pip_import(
    name = "py3_deps",
    compile = False,
    python_interpreter = "python3",
    requirements = "//3rdparty/python3:requirements.transitive.txt",
)

load(
    "@py3_deps//:requirements.bzl",
    pip3_install = "pip_install",
)

pip3_install()

BUILD

config_setting(
    name = "alt_pip",
    define_values = {
        "python_platform": "linux",
    },
)


alias(
    name = "grpc",
    actual = select({
        "//:alt_pip": "@py3_deps_manylinux2010_x86_64//:grpcio",
        "//conditions:default": "@py3_deps//:grpcio",
    }),
)

The issue I'm running into however is that the pip install that "goes first" wins and the subsequent install already sees the packages as downloaded.

Is there any documentation you can provide to help here?

Please document example of namespace handling

Hey @ali5h, I've got an instance of the google.* and azure.* namespace clash issue (bazelbuild/rules_python#93 bazelbuild/rules_python#55). You've indicated that you fixed it, but the problem still persists in v2.1.1, can you give an idea of what that fix looks like?

bazelbuild/rules_python#93 (comment)

i fixed this issue in https://github.com/ali5h/rules_pip

as well bazelbuild/rules_python#93 (comment)

fixed the issues, plz use latest release, thanks for the feedback

I'm using your latest release, and I'm still seeing the issue. Maybe there's additional text I've missed describing what else to do to activate this repair.

If you see https://github.com/chickenandpork/bazel-pip-import-namespace, it represents the issue I'm having. Note that transitory dependencies were tried as well, then commented back out in the BUILD file.

bazel test //... repro the error.

I believe this follows the instructions you've shared. Can you see what else is missing to work around the issue ?

examples/ break with bazel 6.0.0

$ cd examples
$ bazel build //...
Starting local Bazel server and connecting to it...
Analyzing: 8 targets (40 packages loaded, 46 targets configured)
ERROR: /home/jamison/.cache/bazel/_bazel_jamison/aa4d8cfd50269e290d9b8d3fb2845591/external/bazel_tools/platforms/BUILD:84:6: in alias rule @bazel_tools//platforms:linux: Constraints from @bazel_tools//platforms have been removed. Please use constraints from @platforms repository embedded in Bazel, or preferably declare dependency on https://github.com/bazelbuild/platforms. See https://github.com/bazelbuild/bazel/issues/8622 for details.
ERROR: /home/jamison/.cache/bazel/_bazel_jamison/aa4d8cfd50269e290d9b8d3fb2845591/external/bazel_tools/platforms/BUILD:84:6: Analysis of target '@bazel_tools//platforms:linux' failed
ERROR: /home/jamison/code/forks/rules_pip/examples/tests/BUILD:8:15: While resolving toolchains for target //tests:test_import: invalid registered toolchain '@io_bazel_rules_docker//toolchains/docker:default_linux_toolchain': 
ERROR: Analysis of target '//tests:test_import' failed; build aborted: 
INFO: Elapsed time: 4.200s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (46 packages loaded, 265 targets configured)

which succeeds with examples/.bazelversion,

5.4.0

symlink to requirements file not being generated.

We introduced the rules pip in our system a few weeks back, then we started having certain issues, the output error is the following (slightly edited):

INFO: Call stack for the definition of repository 'pypi__3__cython_0_29_13' which is a whl_library (rule definition at /home/ubuntu/.cache/bazel-cache/<custom-output-path>/external/com_github_ali5h_rules_pip/defs.bzl:97:15):
 - /home/ubuntu/.cache/bazel-cache/<custom-output-path>/external/subsystem_import/requirements.bzl:8:5
 - /home/ubuntu/.cache/src/WORKSPACE:308:1
ERROR: An error occurred during the fetch of repository 'pypi__3__cython_0_29_13':
   Not a regular file: /home/ubuntu/.cache/bazel-cache/<custom-output-path>/external/subsystem_import/requirements.txt
ERROR: /home/ubuntu/.cache/src/path/to/target/BUILD:12:1: //path/to/target:python_tool depends on @pypi__3__cython_0_29_13//:pkg in repository @pypi__3__cython_0_29_13 which failed to fetch. no such package '@pypi__3__cython_0_29_13//': Not a regular file: /home/ubuntu/.cache/bazel-cache/<custom-output-path>/external/subsystem_import/requirements.txt
ERROR: Analysis of target '//path/to/target:python_tool' failed; build aborted: no such package '@pypi__3__cython_0_29_13//': Not a regular file: /home/ubuntu/.cache/bazel-cache/<custom-output-path>/external/subsystem_import/requirements.txt

Checking the directory /home/ubuntu/.cache/bazel-cache/<custom-output-path>/external/subsystem_import/ we only have the following files:

BUILD
WORKSPACE
requirements.bzl

Our WORKSPACE looks something like this:

pip_import(
    name = "subsystem_imports",
    python_version = "3",
    requirements = "//path/to/target:requirements.txt",
)

load("@subsystem_imports//:requirements.bzl", subsystem_pip_install = "pip_install")

subsystem_pip_install()

failed to download python packages listed in the requirements3.txt

We are behind a proxy and have https_proxy set already. We are using your WORKSPACE for building, only thing we changed was the python version (3.7). Below is our BUILD file:

load("@piptool_deps_3//:requirements.bzl", "requirement")
py_library(
name = "greeting",
srcs = ["greeting.py"],
deps=[requirement("six"), ]
)

We are getting below errors:

ERROR: An error occurred during the fetch of repository 'pypi__3_7__six_1_12_0':
whl_library failed: (Could not find a version that satisfies the requirement six==1.12.0 (from -c /home/user/.cache/bazel/_bazel_zkf2 07m/95f94b938bcb1f4d30d7174b72ea61e0/external/piptool_deps_3/requirements.txt (line 10)) (from versions: none)
pip._internal.exceptions.DistributionNotFound: No matching distribution found for six==1.12.0 (from -c /home/user/.cache/bazel/_bazel_zkf 207m/95f94b938bcb1f4d30d7174b72ea61e0/external/piptool_deps_3/requirements.txt (line 10))

ruamel.yaml.clib has a non-empty namespace_packages.txt but no files in those packages

I don't know if ruamel.yaml.clib is making a mistake or not, so maybe this issue can be closed and I'll open one on ruamel.yaml's repo. It works normally so I'd guess it's on rules_pip to support it?

Anyway, ruamel.yaml.clib has a top-level C extension (_ruamel_yaml.so) that's used at the top level (from _ruamel_yaml import ...). It also has a namespace_packages.txt with two lines, ruamel and ruamel.yaml. However, it has no folder ruamel or ruamel/yaml, so src/whl.py:43 in _create_nspkg_init tries to create a file ruamel/__init__.py that fails because ruamel/ doesn't exist.

I'm proposing a PR to fix this, but opening this issue for tracking the details on what's going on and why.

Usage of `python_configure`

Me once again.

I was taking a look at your examples directory and noticed the usage of python_configure in your WORKSPACE file. I have not been making use of this and it seems as though things are working correctly. Just wanted to confirm if this was something I should be making use of.

I've been experimenting with configuring my own python toolchain and this seems possibly related.

Thanks.

grpcio error

When loading requirement("grpcio") I get the following error, do you understand what is going on?

Traceback (most recent call last):
  File "/app/demo/feed/nextbus_feed.binary.runfiles/firestore_s2/demo/feed/nextbus_feed.py", line 6, in <module>
    from google.cloud import firestore
  File "/app/demo/feed/nextbus_feed.binary.runfiles/pypi__36__google_cloud_firestore_1_6_0/google/cloud/firestore.py", line 18, in <module>
    from google.cloud.firestore_v1 import __version__
  File "/app/demo/feed/nextbus_feed.binary.runfiles/pypi__36__google_cloud_firestore_1_6_0/google/cloud/firestore_v1/__init__.py", line 22, in <module>
    from google.cloud.firestore_v1._helpers import GeoPoint
  File "/app/demo/feed/nextbus_feed.binary.runfiles/pypi__36__google_cloud_firestore_1_6_0/google/cloud/firestore_v1/_helpers.py", line 21, in <module>
    import grpc
  File "/app/demo/feed/nextbus_feed.binary.runfiles/pypi__36__grpcio_1_26_0/grpc/__init__.py", line 23, in <module>
    from grpc._cython import cygrpc as _cygrpc
ImportError: cannot import name 'cygrpc'

New release planned?

The last release was 3.0.0 in Dec of 2020, but there have been ~20 PR's merged since then. If no releases are planned, that's fine too, but a pinned issue (like this one) with the project's plan to stop releases is a good heads up for folks

Feature request: allow custom prepending to repo name

Some python packages look for rather hard coded strings for search if a package is a pip module:
https://github.com/uqfoundation/dill/blob/master/dill/_dill.py#L1308

Currently the name of the repository is created like this: https://github.com/ali5h/rules_pip/blob/master/src/piptool.py#L75

It would be nice if this could be made customizable with the default as pypi but allowing other strings while calling pip_import.

Feature request: Allow custom BUILD file content to be appended to generated BUILD.

For some python modules that depend on shared libraries written in C++ such as Tensorflow sometimes it is necessary to allow declaration of targets a bit differently. Or to add additional targets such as filegroups. Currently this is not customizable through rules_pip. It would be nice to allow this using an additional parameter to the pip_import rule.

pip compile permission error

I've been encountering a permission error during my builds when the pip_import attribute compile is set to True. I see errors such as Permission denied: '/usr/local/lib/python2.7/... and recommendations to use the --user flag.

While --user seems to work for the pip_install step there doesn't seem to be a way to provide this flag to pip-compile. Digging in it seems newer versions of pip-compile provide a --pip-args flag. I was wondering if you were considering upgrading to a newer version and exposing this through pip_import? Or have another suggestion?

I tried investigating a bit locally (macOS) but have run into trouble running the unit tests for the examples. I'm seeing the following error:

ERROR: no such package '@piptool_deps_tests_3//': pip_import failed:  (src/main/tools/process-wrapper-legacy.cc:58: "execvp(python3.7, ...)": No such file or directory

I'd be happy to work on this further if you could provide some guidance.

Thanks!

Bazel Python rules maintainership collaboration

👋 maintainer of github.com/bazelbuild/rules_python here.

I reached to @ali5h on the Bazel community Slack workspace, but I'm not sure they're hanging around there.

I think it would be good for the maintainers in rules_python to be in touch regularly with the maintainers in this repo, and work together to make there be 'one way' to do Python pip stuff in Bazel.

Working together would be particularly useful in situations related to this issue, bazelbuild/rules_python#395, where this repository has some functionality that's useful and not present in rules_python.

rules_pip should dedup pip dependencies

Using rules_pip against pytest-flask will result in error like

ERROR: /root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/pypi__2_7__pytest_flask_0_14_0/BUILD:6:1: Label '@pypi__2_7__pytest_3_10_1//:pkg' is duplicated in the 'deps' attribute of rule 'pkg'

This is because there're duplicate dependencies.

package(default_visibility = ["//visibility:public"])

load("@pip_libs//:requirements.bzl", "requirement")

py_library(
    name = "pkg",
    srcs = glob(["**/*.py"]),
    data = glob(["**/*"], exclude = [
        "**/*.py",
        "**/*.pyc",
        "**/* *",
        "BUILD",
        "WORKSPACE",
        "bin/*",
        "__pycache__",
    ]),
    # This makes this directory a top-level in the python import
    # search path for anything that depends on this.
    imports = ["."],
    deps = [
        requirement("pytest"),requirement("Flask"),requirement("Werkzeug"),requirement("pytest")
    ],
)

remove par files

just use python, by preparing correct environment in the repository rule

Support for URL requirements

Thanks for creating and sharing your rules!

Do you plan to support URL requirements?

E.g., Using a URL requirement:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile requirements.in
#
...
https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz
...

And calling pip_import results in:

ERROR: no such package '@py//': pip_import failed:  (Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/private/var/tmp/_bazel_jcb/6b822ab9db83c06cf3104c3b5c927ad9/external/com_github_ali5h_rules_pip/tools/piptool.par/__main__.py", line 184, in <module>
  File "/private/var/tmp/_bazel_jcb/6b822ab9db83c06cf3104c3b5c927ad9/external/com_github_ali5h_rules_pip/tools/piptool.par/__main__.py", line 141, in main
  File "/private/var/tmp/_bazel_jcb/6b822ab9db83c06cf3104c3b5c927ad9/external/com_github_ali5h_rules_pip/tools/piptool.par/__main__.py", line 48, in as_tuple
TypeError: Expected a pinned InstallRequirement, got https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz (from -r /private/var/tmp/_bazel_jcb/6b822ab9db83c06cf3104c3b5c927ad9/external/py/requirements.txt (line 22))

Unfortunately, the spacy language models can only be installed via a custom script or URL requirement. https://spacy.io/usage/models#models-download

Bazel build fails due to bad pip cache

Environments can get into a bad state by interrupted pip installs or other corruption of the pip wheel, which leads to all subsequent builds failing with errors similar to:

File "/home/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/f1d1ae8a5075539a72e45ef6b412b203/external/com_github_ali5h_rules_pip/third_party/py/pip/_internal/metadata/base.py", line 321, in as_zipfile
--
  | return zipfile.ZipFile(self.location, allowZip64=True)
  | File "/usr/lib/python3.8/zipfile.py", line 1269, in __init__
  | self._RealGetContents()
  | File "/usr/lib/python3.8/zipfile.py", line 1336, in _RealGetContents
  | raise BadZipFile("File is not a zip file")
  | zipfile.BadZipFile: File is not a zip file
  | Traceback (most recent call last):
  | File "/home/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/f1d1ae8a5075539a72e45ef6b412b203/external/com_github_ali5h_rules_pip/src/whl.py", line 401, in <module>
  | main()
  | File "/home/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/f1d1ae8a5075539a72e45ef6b412b203/external/com_github_ali5h_rules_pip/src/whl.py", line 262, in main
  | pkg = install_package(args.package, args.directory, pip_args)
  | File "/home/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/f1d1ae8a5075539a72e45ef6b412b203/external/com_github_ali5h_rules_pip/src/whl.py", line 108, in install_package
  | dist_info = glob.glob(os.path.join(directory, "*.dist-info"))[0]
  | IndexError: list index out of range
  | )

Internally we've wrapped the pip install with a retry and removal of the package from the local pip cache; wanted to see if there was interest in having the change upstreamed. @ali5h

Support bazel python toolchain() or py_runtime_pair()

google namespace import failing

I am able to recreate an issue that we are seeing in our builds by running rules_pip/examples/tests:

$ bazel test //tests:all
Starting local Bazel server and connecting to it...
INFO: Analyzed target //tests:test_import (64 packages loaded, 4492 targets configured).
INFO: Found 1 test target...
FAIL: //tests:test_import (see /private/var/tmp/_bazel_jcb/13f30459fa9eb9eb617e119869e3a3af/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/tests/test_import/test.log)
INFO: From Testing //tests:test_import:
==================== Test output for //tests:test_import:
============================= test session starts ==============================
platform darwin -- Python 3.7.6, pytest-5.3.2, py-1.8.0, pluggy-0.13.1
rootdir: /private/var/tmp/_bazel_jcb/13f30459fa9eb9eb617e119869e3a3af/sandbox/darwin-sandbox/1/execroot/__main__/bazel-out/darwin-fastbuild/bin/tests/test_import.runfiles
collected 2 items

tests/test_import.py F.                                                  [100%]

=================================== FAILURES ===================================
_________________________________ test_import __________________________________

    def test_import():
        # just try to import some packages that use namespace
>       import google.cloud.language
E       ModuleNotFoundError: No module named 'google.cloud'

tests/test_import.py:3: ModuleNotFoundError
========================= 1 failed, 1 passed in 2.29s ==========================
================================================================================
Target //tests:test_import up-to-date:
  bazel-bin/tests/test_import
INFO: Elapsed time: 133.565s, Critical Path: 5.24s
INFO: 2 processes: 2 darwin-sandbox.
INFO: Build completed, 1 test FAILED, 6 total actions
//tests:test_import                                                      FAILED in 3.2s
  /private/var/tmp/_bazel_jcb/13f30459fa9eb9eb617e119869e3a3af/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/tests/test_import/test.log

INFO: Build completed, 1 test FAILED, 6 total actions

The output of bazel version:

Build label: 2.0.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Dec 19 12:33:30 2019 (1576758810)
Build timestamp: 1576758810
Build timestamp as int: 1576758810

Any ideas?

Numpy dependency issue

Hi,

We are trying to use this with py_binary in order to build a zip file to run inside a docker container. After many days of different problems regarding dependencies not being found inside the docker container, we found your rules and gave them a try and they fixes almost all of them. The only problem that we have left now is something to do with numpy. We would be glad if you could have a quick look.

Our WORKSPACE (just the relevent part):

http_archive(
    name = "com_github_ali5h_rules_pip",
    strip_prefix = "rules_pip-2.0.1",
    sha256 = "befa5df35a7cd6e8f230e50f66128b468b0d7fdd9d960090701efc40c349b42e",
    urls = ["https://github.com/ali5h/rules_pip/archive/2.0.1.tar.gz"],
)

load("@com_github_ali5h_rules_pip//:defs.bzl", "pip_import")

pip_import(
   name = "pip_dependencies",
   requirements = "//tools/dependencies:python_requirements.txt",
#   compile = False,
   python_interpreter = "python3",
   compile = True
)

load("@pip_dependencies//:requirements.bzl", "pip_install")
pip_install([
#    "--only-binary",
#    ":all",
])

Our build:

package(default_visibility = ["//visibility:public"])

load("@pip_dependencies//:requirements.bzl", "requirement")
load(
    "@io_bazel_rules_docker//container:container.bzl",
    "container_push",
    "container_pull",
    "container_image"
)
py_binary(
    name = "index",
    srcs = [
        "index.py",
    ],
    data = glob([
        "static/**/*.*",
        "*.yaml",
    ]),
    deps = [
        requirement("setuptools"),
        requirement("fastai"),
        requirement("pandas"),
        requirement("flask"),
   ],
   srcs_version = "PY3"
)


py_test(
    name = "index_test",
    srcs = ["index_test.py"],
    python_version = "PY3",
    srcs_version = "PY3",
    deps = []
)


container_image(
    name = "project",
    workdir = "/home/project",
    directory = "/home/project",
    cmd = ["index.zip"],
    entrypoint = ["python3"],
    base = "@python_base//image",
    ports = ["5000"],
    files = [
        ":index",
    ]
)

We run bazel run //apps/project:project --build_python_zip in order to build the project image. then we run the built image in docker and get:


IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.7 from "/usr/local/bin/python3",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.17.4" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: No module named 'numpy.core._multiarray_umath'

We don't implicitly use numpy, but one of our dependencies does.
Our tools/dependencies/python_requirements.txt:

click==7.0
numpy==1.17.4
pymongo==3.9.0
flask==1.1.1
PyYAML==5.1.2
pprint==0.1
requests==2.22.0
pika==1.1.0
bs4==0.0.1
dnspython==1.16.0
pandas==0.24.2
setuptools==42.0.2
fastai==1.0.59
torch==1.3.1
torchtext==0.4.0

We saw that in the code you have some implicit reference to numpy, so we think that maybe we are not using something as intended. Could you please assist?

AttributeError: module 'pip._internal.cli' has no attribute 'cmdoptions'

Any ideas on why this happens in a minimal docker container?

INFO: Call stack for the definition of repository 'py_deps' which is a pip_import (rule definition at /root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/defs.bzl:49:14):                                                      
 - /stuff/third_party/python/deps.bzl:14:5
 - /stuff/WORKSPACE:92:1
ERROR: An error occurred during the fetch of repository 'py_deps':
   pip_compile failed:  (Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/src/compile.py", line 3, in <module>                                                                                                                                
    from piptools.scripts import compile
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/scripts/compile.py", line 11, in <module>                                                                                                   
    from .._compat import install_req_from_line, parse_requirements
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/_compat/__init__.py", line 7, in <module>                                                                                                   
    from .pip_compat import (
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/_compat/pip_compat.py", line 54, in <module>                                                                                                
    cmdoptions = do_import("cli.cmdoptions", old_path="cmdoptions")
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/_compat/pip_compat.py", line 37, in do_import                                                                                               
    return getattr(imported, package)
AttributeError: module 'pip._internal.cli' has no attribute 'cmdoptions'
)
ERROR: no such package '@py_deps//': pip_compile failed:  (Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/src/compile.py", line 3, in <module>                                                                                                                                
    from piptools.scripts import compile
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/scripts/compile.py", line 11, in <module>                                                                                                   
    from .._compat import install_req_from_line, parse_requirements
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/_compat/__init__.py", line 7, in <module>                                                                                                   
    from .pip_compat import (
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/_compat/pip_compat.py", line 54, in <module>                                                                                                
    cmdoptions = do_import("cli.cmdoptions", old_path="cmdoptions")
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/_compat/pip_compat.py", line 37, in do_import                                                                                               
    return getattr(imported, package)
AttributeError: module 'pip._internal.cli' has no attribute 'cmdoptions'
)
ERROR: no such package '@py_deps//': pip_compile failed:  (Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/src/compile.py", line 3, in <module>                                                                                                                                
    from piptools.scripts import compile
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/scripts/compile.py", line 11, in <module>                                                                                                   
    from .._compat import install_req_from_line, parse_requirements
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/_compat/__init__.py", line 7, in <module>                                                                                                   
    from .pip_compat import (
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/_compat/pip_compat.py", line 54, in <module>                                                                                                
    cmdoptions = do_import("cli.cmdoptions", old_path="cmdoptions")
  File "/root/.cache/bazel/_bazel_root/5d28f597b39547cacd9fe873932105d8/external/com_github_ali5h_rules_pip/third_party/py/piptools/_compat/pip_compat.py", line 37, in do_import                                                                                               
    return getattr(imported, package)
AttributeError: module 'pip._internal.cli' has no attribute 'cmdoptions'
)

Update to pip 20.x

Is there any plan to update third_party/py/pip to 20.x version?