jq does not properly install its peer dependencies (and relies on a third-party library in setup.py) about jq.py HOT 12 CLOSED

The build system dependencies are specified in pyproject.toml according to PEP 518. What version of pip are you using? Do you have the same issue with the latest version of pip?

from jq.py.

@mwilliamson thanks for the swift reply!

bin/pip --version
pip 21.1.1 from /Users/timo/workspace/wtf/lib/python3.8/site-packages/pip (python 3.8)

bin/python --version
Python 3.8.10

from jq.py.

Could you post the output when you attempt to install the library?

from jq.py.

@mwilliamson sure:

 bin/pip install jq
Collecting jq
  Using cached jq-1.2.1-cp38-cp38-macosx_10_9_x86_64.whl (368 kB)
Installing collected packages: jq
Successfully installed jq-1.2.1
WARNING: You are using pip version 21.1.1; however, version 21.3 is available.
You should consider upgrading via the '/Users/timo/workspace/wtf/bin/python3 -m pip install --upgrade pip' command.

from jq.py.

I'm not sure I understand the issue. It seems like the installation succeeded?

from jq.py.

@mwilliamson yeah. the installation seems to work fine. Though, the package just did not install its install_requires peer dependencies. Therefore the requests library is not installed in your Python environment. When you then execute the code in the jq library that uses requests:

https://github.com/mwilliamson/jq.py/blob/master/setup.py#L16

you will get a traceback that the requests library is missing and it will fail.

If you have the requests library already installed or another package that you use already declares it properly, you will not notice anything.

Though, it is important that Python packages properly declare their dependencies and also install them.

In our case it fails when we try to build a Docker container that has jq as a dependency in the software stack it builds within the Docker container:

https://community.plone.org/t/error-while-building-plone-docker-image-on-mac-m1/14008

I think the right way to solve this issue is to remove the requests dependency altogether and replace it with a urllib3 call. Even if we fix this particular problem, relying on a third-party library in your setup.py is very fragile and can easily break in other circumstances.

from jq.py.

Just a follow up. I upgraded my pip version to a version that supports pyproject.toml:

https://pip.pypa.io/en/stable/news/#v21-2

The problem still persists though.

Maybe packages that rely on pyproject.toml should declare this in their setup.py and fail hard on old pip versions. Something like this line could help:

https://github.com/plone/plone.restapi/blob/master/setup.py#L6

from jq.py.

The requests library is only a dependency during setup/installation though. It is not used when you import jq as far as I see.

In most cases, pip will use one of the many wheels of this package. In that case, requests is not even needed during install.

By default I get this on my Mac (10.15), which is very fast:

$ bin/pip install jq
Collecting jq
  Downloading jq-1.2.1-cp39-cp39-macosx_10_9_x86_64.whl (371 kB)
Installing collected packages: jq
Successfully installed jq-1.2.1

Compare this with a forced non-binary install:

$ bin/pip install --no-binary :all: jq
Collecting jq
  Downloading jq-1.2.1.tar.gz (72 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: jq
  Building wheel for jq (PEP 517) ... done
  Created wheel for jq: filename=jq-1.2.1-cp39-cp39-macosx_10_15_x86_64.whl size=371235 sha256=e1cd04048c7463a2df20b8929772fcacc32469849ee1af750ea8cd31b381cb0f
  Stored in directory: /Users/maurits/Library/Caches/pip/wheels/56/c8/df/022dbdb5421ddcc1a3c3a2a8df1da33e2703a30d19c3f28dd0
Successfully built jq
Installing collected packages: jq
Successfully installed jq-1.2.1

This takes much longer, maybe a minute or so. So it is very good that there are so many wheels available by default already. But apparently not for the newer Mac M1 machines.

But both, the result is that import requests fails, but it is no problem because import jq works.

from jq.py.

To reproduce the problem though:

$ python3.9 -mvenv .
$ bin/pip install pip<20
$ bin/pip install --no-binary :all: jq

Wow, this even fails for me because setuptools cannot be found. :-)
This works:

$ bin/pip install --no-binary jq,requests jq

I guess on some platforms this could fail when requests itself cannot be installed for whatever reason via source, and there is no wheel available on PyPI.

from jq.py.

Actually, the first pip version that supports pyproject.toml is 19.0. But even with pip 18.1, locally it finds and temporarily installs the build dependencies just fine. I am not sure how it knows how to do this.

Anyway, not sure what jq could do to improve this, except building and uploading even more wheels.

from jq.py.

Though, it is important that Python packages properly declare their dependencies and also install them.

As @mauritsvanrees says, requests is not a runtime dependency of jq. You shouldn't need to execute setup.py to use jq as a Python library once it's installed.

I think the right way to solve this issue is to remove the requests dependency altogether and replace it with a urllib3 call

I'm not sure that would solve the problem: so far as I know, urllib3 is not in the standard library, same as requests. Using urllib.request instead would be an option.

Just a follow up. I upgraded my pip version to a version that supports pyproject.toml:

https://pip.pypa.io/en/stable/news/#v21-2

The problem still persists though.

If installation fails on recent versions of pip, then that seems more problematic. Do you have an example of the command and output when this happens?

Actually, the first pip version that supports pyproject.toml is 19.0. But even with pip 18.1, locally it finds and temporarily installs the build dependencies just fine. I am not sure how it knows how to do this.

I think pip 18.0 added support for PEP 518 and therefore the way in which jq uses pyproject.toml. pip 19.0 added support for PEP 517, which I don't think is needed by jq.

from jq.py.

The dependencies are part of the sdist now, so this shouldn't be an issue any more.

from jq.py.