[FreeBSD] ModuleNotFoundError: No module named 'versioningit' about mqttwarn HOT 37 CLOSED

from mqttwarn.

Maybe we can discover other Python packages already using build-time dependencies like setuptools-scm or versioningit, (or versioneer or vcversioner), in order to learn how they are doing it to overcome the Git vs. Tarball barrier.

from mqttwarn.

Rest assured we will definitively not go that route, and keep proper release and bundle management. The user should surely not be concerned to pull or clone from any Git repository at all. But it's a different thing for package builders or distributors. Apologies that this currently causes troubles for FreeBSD.

from mqttwarn.

pep517 instead of distutils works.

from mqttwarn.

Thanks also for your efforts to bring versioningit to the FreeBSD ports tree, so people have more choices other than setuptools-scm.

/cc @jwodder

from mqttwarn.

Committed:

• https://www.freshports.org/sysutils/py-mqttwarn/
• https://www.freshports.org/devel/py-versioningit/

Thank you

from mqttwarn.

Dear Dan,

thanks for taking care. We recently introduced the Python build dependency package versioningit, because it makes releasing and version bumping so much easier for us. The benefit is, for example, reflected by now being able to tell OCI PR-builds, or any other development builds apart ¹.

$ docker run --rm ghcr.io/jpmens/mqttwarn-standard:pr-645 mqttwarn --version
mqttwarn 0.34.0.post10+gc972d99.d20230428

On the build environment side, this means it introduces two new prerequisites: The git program, and the versioningit Python package. Note that we also upgrade pip and install the wheel package, but that's probably optional and depends on your taste or corresponding packaging policies.

https://github.com/jpmens/mqttwarn/blob/a3746619b6cb3ca060c4ba73a44ed747a3331c59/Dockerfile#L21
https://github.com/jpmens/mqttwarn/blob/a3746619b6cb3ca060c4ba73a44ed747a3331c59/Dockerfile#L35-L36

Will you be able to provision your build environment correspondingly by adding those two essential dependencies?

With kind regards,
Andreas.

from mqttwarn.

I will also check if and how versioningit can be officially declared as a build-time dependency. I know it is possible by using a more modern project metadata definition based on pyproject.toml ¹, but I don't know if it will mix well together with the oldschool setup.py we are still using here.

Let's just try it!

from mqttwarn.

Oh, we do have everything in place already, as per PEP 517 and PEP 518.

https://github.com/jpmens/mqttwarn/blob/08c14c83ab117b9890de56781df444a7d35e6766/pyproject.toml#L5-L17

May I ask how your build procedure for FreeBSD looks like, so that I can reproduce and possibly fix this?

from mqttwarn.

I am the FreeBSD port maintainer for https://www.freshports.org/sysutils/py-mqttwarn/ and I'm attempting to upgrade the port from 0.31.0 to 0.34.0

The build procedure is in this Makefile: https://cgit.freebsd.org/ports/tree/sysutils/py-mqttwarn

This is the latest build (using poudriere): https://services.unixathome.org/poudriere/build.html?mastername=131amd64-dvl&build=2023-04-28_20h07m27s

This is the log from that build: https://services.unixathome.org/poudriere/data/131amd64-dvl/2023-04-28_20h07m27s/logs/errors/py39-mqttwarn-0.34.0.log

I hope that last link answers everything.

from mqttwarn.

Hi again,

thanks for sharing those valuable resources. Do you think adding a BUILD_DEPENDS section, minimally including git and python-versioningit, and optionally also including python-wheel, could help here?

With kind regards,
Andreas.

from mqttwarn.

Hi Dan,

while you are working on refreshing the FreeBSD port of mqttwarn (thank you so much!), may I ask you if you could also adjust one or both files containing project metadata ¹² to list both URLs to the repository ³ and the documentation ⁴, which only recently has been added.

On the other hand, I will be happy to update the documentation section about installing mqttwarn ⁵ with information about FreeBSD, if you tell me how it works there. Is it probably just pkg install py-mqttwarn?

With kind regards,
Andreas.

from mqttwarn.

Yes, I can do that.

Is https://pypi.org/project/versioningit/ the module you're using? That's what I'm going to try porting.

from mqttwarn.

Exactly, that is the correct package. Thank you!

from mqttwarn.

Yeah, this won't work at all for FreeBSD as far as I can tell. It's expecting to be building from a git repository. Well, it's not. It's building from a tarball.

https://services.unixathome.org/poudriere/data/131amd64-dvl/2023-04-29_12h09m26s/logs/errors/py39-mqttwarn-0.34.0.log

===>   py39-mqttwarn-0.34.0 depends on package: py39-versioningit>=0 - found
===>   Returning to build of py39-mqttwarn-0.34.0
===>   py39-mqttwarn-0.34.0 depends on package: py39-setuptools>=63.1.0 - found
===>   py39-mqttwarn-0.34.0 depends on file: /usr/local/bin/python3.9 - found
===========================================================================
=======================<phase: lib-depends    >============================
===========================================================================
=>> Recording filesystem state for prebuild... done
=======================<phase: configure      >============================
===>   py39-mqttwarn-0.34.0 depends on package: py39-versioningit>=0 - found
===>   py39-mqttwarn-0.34.0 depends on package: py39-setuptools>=63.1.0 - found
===>   py39-mqttwarn-0.34.0 depends on file: /usr/local/bin/python3.9 - found
===>  Configuring for py39-mqttwarn-0.34.0
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/versioningit/git.py", line 146, in ensure_is_repo
    self.read(
  File "/usr/local/lib/python3.9/site-packages/versioningit/git.py", line 178, in read
    return readcmd("git", *args, cwd=str(self.path), **kwargs)
  File "/usr/local/lib/python3.9/site-packages/versioningit/util.py", line 71, in readcmd
    s = runcmd(*args, stdout=subprocess.PIPE, text=True, **kwargs).stdout
  File "/usr/local/lib/python3.9/site-packages/versioningit/util.py", line 66, in runcmd
    return subprocess.run(arglist, **kwargs)
  File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'rev-parse', '--is-inside-work-tree']' returned non-zero exit status 128.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/versioningit/core.py", line 257, in run
    description = self.do_vcs()
  File "/usr/local/lib/python3.9/site-packages/versioningit/core.py", line 321, in do_vcs
    description = self.vcs(project_dir=self.project_dir)
  File "/usr/local/lib/python3.9/site-packages/versioningit/methods.py", line 162, in __call__
    return self.method(params=self.params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/versioningit/git.py", line 225, in describe_git
    repo.ensure_is_repo()
  File "/usr/local/lib/python3.9/site-packages/versioningit/git.py", line 156, in ensure_is_repo
    raise NotVCSError(f"{self.path} is not in a Git repository")
versioningit.errors.NotVCSError: /wrkdirs/usr/ports/sysutils/py-mqttwarn/work-py39/mqttwarn-0.34.0 is not in a Git repository

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/versioningit/core.py", line 539, in get_version_from_pkg_info
    Path(project_dir, "PKG-INFO").read_text(encoding="utf-8")
  File "/usr/local/lib/python3.9/pathlib.py", line 1266, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/usr/local/lib/python3.9/pathlib.py", line 1252, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/usr/local/lib/python3.9/pathlib.py", line 1120, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/wrkdirs/usr/ports/sysutils/py-mqttwarn/work-py39/mqttwarn-0.34.0/PKG-INFO'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/versioningit/hook.py", line 23, in setuptools_finalizer
    report = vgit.run(write=True, fallback=True)
  File "/usr/local/lib/python3.9/site-packages/versioningit/core.py", line 280, in run
    version=get_version_from_pkg_info(self.project_dir)
  File "/usr/local/lib/python3.9/site-packages/versioningit/core.py", line 542, in get_version_from_pkg_info
    raise NotSdistError(f"{project_dir} does not contain a PKG-INFO file")
versioningit.errors.NotSdistError: /wrkdirs/usr/ports/sysutils/py-mqttwarn/work-py39/mqttwarn-0.34.0 does not contain a PKG-INFO file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "setup.py", line 176, in <module>
    setup(
  File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 139, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 476, in __init__
    _Distribution.__init__(
  File "/usr/local/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 275, in __init__
    self.finalize_options()
  File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 900, in finalize_options
    ep(self)
  File "/usr/local/lib/python3.9/site-packages/versioningit/hook.py", line 28, in setuptools_finalizer
    raise RuntimeError(
RuntimeError: 
versioningit could not find a version for the project in /wrkdirs/usr/ports/sysutils/py-mqttwarn/work-py39/mqttwarn-0.34.0!

You may be installing from a shallow clone, in which case you need to unshallow it first.

Alternatively, you may be installing from a Git archive, which is not supported by default.  Install from a git+https://... URL instead.


*** Error code 1

Stop.
make: stopped in /usr/ports/sysutils/py-mqttwarn

from mqttwarn.

Yeah, this won't work at all for FreeBSD as far as I can tell. It's expecting to be building from a git repository. Well, it's not. It's building from a tarball.

Oh, I see. Thank you for reporting back. I will see what we can do here, leaving FreeBSD behind would be very sad.

from mqttwarn.

So many tools are expecting the user to install from (or run from) a git clone - please don't go that route.

from mqttwarn.

I've just discovered those two discussions, which basically reflect about the same thing, how to bring downstream package building into the PEP 517 world, only related to Debian instead of FreeBSD. Maybe I can figure out from that what's missing yet to make that work well for mqttwarn.

• jwodder/versioningit#28
• jwodder/versioningit#29

from mqttwarn.

Maybe we can discover other Python packages already using build-time dependencies like setuptools-scm [...]

Tripping into pep 517 and freebsd, I've just discovered the OCRmyPDF project, which apparently uses setuptools-scm within a PEP517-style project metadata configuration already.

Following up on that, I discovered ocrmypdf/OCRmyPDF#375, and Installing pikepdf on FreeBSD. They refer to two installation variants:

pkg install py38-pikepdf

vs.

pkg install python3 py38-lxml py38-pip py38-pybind11 qpdf
pip install --user pikepdf

It is not clear to me yet how they are doing it, or if the second variant would be obligatory, which would be sad. However, reading ocrmypdf/OCRmyPDF#493 or ocrmypdf/OCRmyPDF#810, it looks like they are well maintaining a proper release pipeline based on PEP 517 and FreeBSD.

from mqttwarn.

Tripping into pep 517 and freebsd [...]
[...] maintaining a proper release pipeline based on PEP 517 and FreeBSD.

I haven't digested it yet, but this resource will surely help to learn about that very subject.

Introduction

PEP-517 introduces a new build and install paradigm for Python packages that the ports framework, as of Q2 2022, does not support. This page explains why such support is needed and discusses how to get there.

In a nutshell, PEP-517 is a minimal format/interface/standard for Python package build systems to generate wheels from source trees or distributions, which can then be installed into one's Python distribution (site-packages).

Why support PEP-517

Packages from CHEESESHOP are increasingly not shipping with setup.py that USE_PYTHON=distutils executes to build and install the package in staging. Instead, they follow PEP-517 with a pyproject.toml, similar to Rust/Cargo packages' Cargo.toml; any additional files related to building and installing are specific to the chosen build systems in pyproject.toml. setuptools no longer plays the central distutils role of building and installing; it is now one of many build backends packages can specify.

-- https://wiki.freebsd.org/Python/PEP-517

from mqttwarn.

py-mqttwarn currently uses USE_PYTHON=autoplist concurrent distutils. According to the documentation references above, this may become USE_PYTHON=autoplist pep517, effectively replacing distutils, because it will be deprecated anyway.

Additionally, distutils is deprecated with removal planned for Python 3.12. We need to start future-proofing the framework now, not least to avoid surprises later, but to gradually allow existing ports to transition and add new ports following PEP-517.

However, I don't understand yet how or if that will also manage the Git vs. Tarball issue well, as this documentation page does not yet talk about the feature we are using here, omitting the version attribute from the package metadata completely, and letting versioningit figure it out at build-time, which, as demonstrated, currently needs building from a Git repository instead of a Tarball.

For now, I am happy that we have been able to at least discover that this indeed is a topic in Python/FreeBSD packaging, and that we are not chasing ghosts. I will continue reporting about further discoveries here.

from mqttwarn.

Many other details are being discussed over here. However, there are no specific solutions for FreeBSD on the very topic of overcoming the Git vs. Tarball impedance mismatch.

• PEP 517 workflow for distributions
• Proposal for dynamic metadata plugins

from mqttwarn.

Also, after reading the documentation of versioningit a bit more thoroughly, I discovered those resources which may be interesting to explore.

1. The "git-archive" method is a variation of the "git" method that also supports determining the version when installing from a properly-prepared Git archive.
2. The tool.versioningit.onbuild and tool.versioningit.write configuration sections, allowing to insert the project version and/or other fields into built project trees when building an sdist or wheel.

from mqttwarn.

[...] allowing to insert the project version and/or other fields into built project trees when building an sdist or wheel.

@dlangille: Would that actually be an option, to build from the Python sdist tarball in any way? After reading the details about FreeBSD packaging for py-mqttwarn more thoroughly, it is still not 100% clear to me how the tarball is obtained, which is used for building. Apologies that I am not that familiar with this process on FreeBSD, so I am humbly asking for further education on this detail.

I think if we could write the version into that tarball somehow, and leverage it from there, it would resolve this issue.

from mqttwarn.

it is still not 100% clear to me how the tarball is obtained

Answering myself: All right, I see that the SHA256 digest at ¹ equals the one of the corresponding version at PyPI ², so I recognize it is just this tarball, which seeds the build process.

from mqttwarn.

Correct.

from mqttwarn.

I've verified it should work well in general, within a fresh Python virtualenv where I've only installed versioningit, and a recent mqttwarn tarball obtained from PyPI.

Prepare

python3 -m venv .venv
source .venv/bin/activate
pip install versioningit
wget --no-clobber https://files.pythonhosted.org/packages/6d/12/73bb680883c53510c760dec432a13240bbc26569b248e75a46100d307d95/mqttwarn-0.34.0.tar.gz
tar -xzf mqttwarn-0.34.0.tar.gz
cd mqttwarn-0.34.0

Build using deprecated method

python setup.py install
Finished processing dependencies for mqttwarn==0.34.0

Build using modern methods

pip install .
Successfully installed mqttwarn-0.34.0

pip install build
python -m build
Successfully built mqttwarn-0.34.0.tar.gz and mqttwarn-0.34.0-py3-none-any.whl

Better safe than sorry

I made sure it all happened outside a Git repository.

$ git describe
fatal: not a git repository (or any of the parent directories): .git

from mqttwarn.

Reading #646 (comment) again, I think this is the root cause:

FileNotFoundError: [Errno 2] No such file or directory: '/wrkdirs/usr/ports/sysutils/py-mqttwarn/work-py39/mqttwarn-0.34.0/PKG-INFO'

If project_dir is not under version control (or if the VCS executable is not installed), versioningit will assume that the directory is an unpacked sdist and will read the version from the PKG-INFO file.

-- versioningit/core.py#L236-L240

On my research, I've read about tools stripping the PKG-INFO file after extracting the tarball, but I can't find it back right now. Can you verify this is not happening on your end? Maybe switching over to USE_PYTHON=autoplist pep517 will do the trick already, and we would be good to go?

from mqttwarn.

Unrelated to my post, I also wanted to share those discussions with you, which report about the current state of the onion with respect to PEP-517 support on FreeBSD. I think it should work well in general, modulo a few edge cases.

• https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722
• https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268893

from mqttwarn.

I know the cause. Me.

The build system was having trouble downloading from pypi:

[15:07 pkg01 dan ~/ports/head/sysutils/py-mqttwarn] % sudo make makesum
===>  License EPL accepted by the user
===>  License EPL accepted by the user
===>   py39-mqttwarn-0.34.0 depends on file: /usr/local/sbin/pkg - found
=> mqttwarn-0.34.0.tar.gz doesn't seem to exist in /usr/home/dan/ports/head/distfiles/.
=> Attempting to fetch https://files.pythonhosted.org/packages/source/m/mqttwarn/mqttwarn-0.34.0.tar.gz

I grabbed the tarball from github.

The two tarballs are not the same, I see now. The pypi tarball has the PKG-INFO file and builds.

https://services.unixathome.org/poudriere/build.html?mastername=131amd64-dvl&build=2023-04-29_14h56m17s

The new release is installed and running here:

[15:05 mqtt01 dan ~] % mqttwarn --version
mqttwarn 0.34.0

from mqttwarn.

Excellent!

from mqttwarn.

That fetch problem seems to be IPv6 related.

I'll update the FreeBSD ports tree soon.

from mqttwarn.

Thank you so much!

from mqttwarn.

Thank you. I am adding a corresponding section about installation on FreeBSD with GH-651.

from mqttwarn.

Following up on my proposal to update the documentation a bit at #646 (comment), I've just submitted a patch at freebsd/freebsd-ports#175, and adjusted GH-651 according to your feedback. I hope you like it.

from mqttwarn.

Dear @dlangille,

I think it will be safe to close this issue. Can I humbly ask you to have a look at freebsd/freebsd-ports#175, as a last step before doing so?

With kind regards,
Andreas.

from mqttwarn.

Thank you for integrating the package metadata / documentation update into the FreeBSD ports tree with https://cgit.freebsd.org/ports/commit/?id=35c70b043178, and for all your other efforts. Closing this now.

from mqttwarn.