lkhq / debspawn Goto Github PK

When I try to build packages with debspawn in source directory, I passed --only binary option to build binary package only, but it still try to invoke dpkg-buildpackage -S to build a source package.

Is there any way to build a binary package without a source package using debspawn ?

My command below:

$ ./debspawn.py --verbose build --only binary jammy .

Thanks in advance :-)

debbuild is a tool that accepts RPM spec files and outputs installable Debian packages as well as the equivalent of ".src.rpm" as ".sdeb" packages.

It'd be great if debspawn supported building Debian packages using debbuild in addition to the traditional dsc packaging styles.

Hi @ximion
I have a question.
How can I set CFLAGS or CXXFLAGS while building the package?
When I try to use debspawn build buster --buildflags='-g' or
debspawn build buster --buildflags='-O1'
I get error dpkg-buildpackage: error: unknown option or argument -O1. Can't I pass -g or -O1 to buildflags?

Thank you!

In the same vein as #9, it would be super awesome to run the autopkgtest suite after the build. (Probably using the autopkgtest-virt-null backend, since we're already inside an ephemeral container, so adding another layer of containerization is probably not worth it).

tl;dr

Would it be possible to allow passing through of nspawn's switches to debspawn?

background

debspawn login allows interaction with the container, f.e. administrative stuff using the --persistent switch. But login doesn't allow to mount a directory from the host. Also, I wasn't able to store any data to the host. Writing to /srv/build inside the container didn't create a file anywhere on the host.

workaround

A hack would be to use debspawn run --build-dir=/opt/here buster /usr/bin/env bash.

benefits

The user could even mount a directory as read-only if it shouldn't be modified during the session.

Running this command;

$ debspawn create green

Produces this output;

╔═════════════════════════════════════════╗
║  Creating new base: green [i386]        ║
╚═════════════════════════════════════════╝
Using mirror: default
Bootstrap suite: green

┌───────────────────┐
│  Bootstrap        │
└───────────────────┘
I: Retrieving InRelease 
I: Checking Release signature
I: Valid Release signature (key id 0D6695569A8253FF9C389F552B4A53F2B41CE072)
E: Invalid Release file, no entry for main/binary-i386/Packages

Environment;
debootstrap 1.0.89
Debian GNU/Linux 9.9 (stretch)

Transferred from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024090

debspawn create unstable fails because it cannot create the builder
user: adduser does no longer seem to be part of the default install.
Either add adduser in include_pkgs in osbase.py, or create the user with
the lower-level tools.

@ximion I don't think the workaround in 222dadb for #26 works for using debos in Debian Bookworm containers:

$ debspawn create bookworm
$ debspawn run --allow=kvm,read-kmods --cachekey=bookworm-resolved --external-command --init-command=prepare_container.sh --build-dir="$PWD" --artifacts-out="$PWD" bookworm build_image.sh

I get the following error while preparing the container:

┌─────────────────────────────┐
│  Preparing container        │
└─────────────────────────────┘
Ign:1 http://deb.debian.org/debian bookworm InRelease
Ign:1 http://deb.debian.org/debian bookworm InRelease
Ign:1 http://deb.debian.org/debian bookworm InRelease
Err:1 http://deb.debian.org/debian bookworm InRelease
  Temporary failure resolving 'deb.debian.org'
Reading package lists...
W: Download is performed unsandboxed as root as file '/var/lib/apt/lists/partial/deb.debian.org_debian_dists_bookworm_InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
E: Failed to fetch http://deb.debian.org/debian/dists/bookworm/InRelease  Temporary failure resolving 'deb.debian.org'
E: Some index files failed to download. They have been ignored, or old ones used instead.
Command `apt-get -uyq -o Dpkg::Options::="--force-confnew" update` failed.
ERROR: Container setup failed.

I was able to get the container to build by following the suggestion in #26 and removing /etc/resolv.conf at the end of prepare_container.sh.

prepare_container.sh

#!/bin/sh

set \
    -o errexit \
    -o nounset

export DEBIAN_FRONTEND=noninteractive

apt-get install --yes --quiet \
        debos \
        kmod \
        udev \
        parted \
        cryptsetup \
        binfmt-support \
        systemd-resolved \
        ca-certificates

build_image.sh

#!/bin/bash

set \
  -o errexit \
  -o nounset \
  -o pipefail

debos \
  --fakemachine-backend=kvm \
  rootfs.yml

rootfs.yml

architecture: amd64

actions:
  - action: debootstrap
    suite: bookworm
    components:
      - main
    mirror: https://deb.debian.org

I'd like to be able to disable the internal apt caching logic with an (off by default!) flag, as I already have apt-cacher-ng setup and have only one cache between apt-cacher-ng and my system apt cache. I've setup debspawn to use the apt proxy provided by apt-cacher-ng, but I'd rather not have duplicate copies of all the debs I'm building between sbuild (because debspawn isn't yet able to run piuparts or autopkgtest), my system deb cache, and debspawn.

I thought I would report this since I wasn't sure how I might fix. I did an update of the distribution and that solved it.

╔═══════════════════════╗
║  Package build        ║
╚═══════════════════════╝
Package: tpmfactoryupd
Version: 1.1.2529.00-1pureos4
Distribution: byzantium
Architecture: amd64

dpkg-source: warning: extracting unsigned source package (/home/jeremiah/packaging/tpmfactoryupd/../tpmfactoryupd_1.1.2529.00-1pureos4.dsc)
dpkg-source: info: extracting tpmfactoryupd in tpmfactoryupd-1.1.2529.00
dpkg-source: info: unpacking tpmfactoryupd_1.1.2529.00.orig.tar.xz
dpkg-source: info: unpacking tpmfactoryupd_1.1.2529.00-1pureos4.debian.tar.xz
Free space in workspace: 380.8GiB
usage: dsrun [-h] [--update] [--no-color] [--no-unicode] [--arch-only]
             [--build-prepare] [--build-run] [--lintian]
             [--buildflags BUILDFLAGS] [--suite SUITE] [--prepare-run]
             [--run-qa]
dsrun: error: unrecognized arguments: --buid=67068
ERROR: Build environment setup failed.

The instructions for building a package in PureOS are to use debspawn with a "landing" argument, like this;

$ debspawn build landing --sign

The "landing" argument to debspawn designates the container image to build in, but if landing doesn't exist one needs to build it, which can be done with debspawn. This has to be done because we get the error;

ERROR: The container image for "landing-amd64" does not exist. Please create it first.

However, the usual debspawn way to create a container fails if you do not already have the name of the container as a script in debootstrap;

$ debspawn create landing
╔════════════════════════════════════════════╗
║  Creating new base: landing [amd64]        ║
╚════════════════════════════════════════════╝
Using mirror: default
Bootstrap suite: landing

┌───────────────────┐
│  Bootstrap        │
└───────────────────┘
E: No such script: /usr/share/debootstrap/scripts/landing

This is on an up to date debootstrap which still has the symlink of landing pointing to green. If the workaround is to symlink landing to byzantium, why not simply build for byzantium with debspawn and skip landing?

The newer versions of debootstrap no longer install packages which are Priority: required but not essential for the buildd variant. This means that passwd will not be installed, and thus, user creation will fail.

This is tracked downstream as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055065.

There are a couple of different ways that I could see us fixing this, all with their pros and cons.

Manually install passwd

This is the simplest option. We could simply unconditionally pass --include=passwd to the invocation of debootstrap. That would leave us in the exact situation we were in before hand. However, this means that we are 1) tracking independently what packages are needed, and 2) putting packages into the container that are technically not necessary for the actual build.

Create the user from outside the container

This is what sbuild does -- it invokes useradd from the host system, not from inside the chroot. This requires more coding work -- it can't live in dsrun -- but prevents us from contaminating the build container with things that are technically not necessary.

systemd magic

The last option would be to use something like the systemd-nspawn user namespace stuff to run the build as a user controlled by systemd. This has a hard dependency on systemd, obviously, but we already have that. In some ways, this is the "cleanest" solution -- we don't ever have to manage the user directly ourselves again -- but also the most complicated from an implementation standpoint. I'm not even 100% sure it's possible, though a read of the documentation seems to imply that it is.

Thoughts?

like sbuild --extra-package=package.deb and --extra-repository=spec.

Description

As this moment, debspawn assumes that a suite is the name for the container. But, in real use cases, there can be situations where users would want to have custom named containers, like unstable-other, testing-extra, which have the main distro as their base but also have some other external repositories configured.

Now I could easily create an unstable container and add the extra repository and persist it with debspawn login --persistent sid. But debspawn does not provide an easy way to rename a container. And debspawn created containers are not listed under machinectl list-images, which could have been another quick interface to rename a container.

So would it be okay to add a debspawn rename command similar to machinectl rename ?

Hello and thanks for developing debspawn. I'm the author of https://salsa.debian.org/federico/conbuilder and I stopped developing it since debspawn is in Debian.
Would you consider adopting the 2-layered build described in the readme if it's useful? It could make starting builds significantly faster.

like sbuild --no-clean-source.

This is useful when used with gbp. So you don't need to install build-depends on host.

debspawn run -h shows:

usage: debspawn run [-h] [--variant VARIANT] [-a ARCH] [--suite SUITE] [--artifacts-out ARTIFACTS_DIR] [--build-dir BUILD_DIR] [--bind-build-dir {y,n,ro,rw}] [--cachekey CACHEKEY] [--init-command INIT_COMMAND]  
                    [-x] [--header HEADER] [--allow ALLOW] [--boot]                                                                                                                                                
                    [name] [command ...]                                                                                                                                                                           
                                                                                                                                                                                                                   
positional arguments:
  name                  The name of the container image (usually a distribution suite name).
  command               The command to run.

While man debspawn-run shows:

SYNOPSIS
       debspawn run [-h|--help] [--variant] [-a|--arch] [--suite] [--artifacts-out] [--build-dir] [--bind-build-dir] [--cachekey] [--init-command] [-x|--external-command] [--header] [--allow] [--boot]
                    {SUITE}

In the man page it is not clear that a command can be passed after {SUITE}, and maybe {SUITE} should be {NAME}?

Recently upgraded from stretch to buster and got this error;

Traceback (most recent call last):
File "/usr/local/bin/debspawn", line 5, in
from debspawn import cli
ModuleNotFoundError: No module named 'debspawn'

I reinstalled debspawn and the error went away.

I'm unable to update images with debspawn, I get this error;

ERROR: Can not update "amber-amd64": The configuration does not exist.

I cannot create images with debspawn either;

/usr/local/bin/debspawn run --external-command --artifacts-out=/srv/artifacts/1111/ amber /home/jeremiah/code/shell/build-pureos.sh produces;

ERROR: Can not run command in "amber-amd64": The base image does not exist.

This is true for 0.2.1 and 0.2.2

hi,

the current ongoing usr-is-merge situation broke my existing debian/sid chroots on bookworm. I attempted to update debootstrap to the version in sid and apparently, if creating a new unstable chroot using debspawn, this error is to be
seen during configuration step:

Failed to mount /etc/localtime (type n/a) on /var/tmp/debspawn/6mqxcr8d/usr/share/zoneinfo/UTC (MS_BIND ""): No such file or directory

this seems to become an issue, as /etc/localtime ends up beeing a regular file within the created chroot and installing
any package that depends on tzdata fails.

So i setup a debian/sid vm and it seems this issue appears with debspawn in debian/sid, too.
Its reproducible via:

debspawn git checkout + change for: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055065

debspawn --version
0.6.3
debspawn create unstable
┌───────────────────┐
│  Configure        │
└───────────────────┘
Failed to mount /etc/localtime (type n/a) on /var/tmp/debspawn/8levserq/usr/share/zoneinfo/UTC (MS_BIND ""): No such file or directory
Hit:1 http://deb.debian.org/debian unstable InRelease
[..]
debspawn run unstable apt install tzdata
[..]
mv: cannot move '/etc/localtime.dpkg-new' to '/etc/localtime': Device or resource busy
dpkg: error processing package tzdata (--configure):
 installed tzdata package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 tzdata
E: Sub-process /usr/bin/dpkg returned an error code (1)

(venv) root@sid:~/debspawn# dpkg -l systemd-container
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name              Version      Architecture Description
+++-=================-============-============-=================================
ii  systemd-container 254.5-1      amd64        systemd container/nspawn tools

as represented during login:

 debspawn login unstable
[..]
Spawning container debian12.localdomain-unstable-buildd-amd64-jsie on /var/tmp/debspawn/er0w107r.
Press Ctrl-] three times within 1s to kill container.
root@sid:/srv# ls -alh /etc/localtime
-rw-r--r-- 1 root root 2.8K Aug  9 14:10 /etc/localtime

using debootstrap to create a regular sid chroot, the file is a symlink:

debootstrap sid sid
[..]
ls -alh sid/etc/localtime 
lrwxrwxrwx 1 root root 27 Nov  2 12:02 sid/etc/localtime -> /usr/share/zoneinfo/Etc/UTC

and spawning this chroot via systemd-nspawn directly:

root@debian12:~# systemd-nspawn -D sid
Spawning container sid on /root/sid.
Press Ctrl-] three times within 1s to kill container.
root@sid:~# ls -alh /etc/localtime 
lrwxrwxrwx 1 root root 41 Nov  2 12:04 /etc/localtime -> ../usr/share/zoneinfo/America/Los_Angeles

as for the created image:

/var/lib/debspawn/images# mkdir foo; tar -xvf unstable-buildd-amd64.tar.zst -C foo
ls -alh foo/etc/localtime 
-rw-r--r-- 1 root root 2.8K Aug  9 14:10 foo/etc/localtime

not quite sure what debspawn does differently in regards to plain systemd-nspawn.

Hi,

If you're not familiar with QEMU user mode, it's like magic! It's a Wine-like wrapper that lets you run foreign executables. It can be used to run foreign chroots, although the statically linked version (qemu-user-static package) is needed so it won't have to resolve symbols outside the container.

This works with systemd-nspawn, as I used to do it by hand. The qemu-user-static package comes with a very minimal debootstrap wrapper that takes care of the voodoo, and can probably be used drop-in.

Please add support for it, or for otherwise allowing one to specify an alternative debootstrap executable.

Hi,

sbuild maintainer here. Today I learned that debspawn exists and as I seek to reduce the list of System Build Tools I would like to find out what debspawn offers that sbuild does not.

For example the first paragraph claims that sbuild uses a plain chroot. True it can do that. But it can also use any environment supported by autopkgtest (like lxc, lxd, qemu, schroot, ssh...) as well as the unshare backend which uses linux user namespaces to do package builds. What advantage does systemd-nspawn have over that? I'm tempted to implement a systemd-nspawn backend and/or a debspawn backend into sbuild but it seems that systemd-nspawn requires super user privileges while the sbuild unshare backend does not (not for the setup, nor for image creation and neither for running the build). Maybe you can point out in the README why a user would prefer to use debspawn over the sbuild unshare backend.

You state that a difference between sbuild and debspawn is unicode handling. Could you point me to the debspawn code that implements this so that I can understand what this paragraph means? Maybe this functionality should be included into sbuild.

You also state that sbuild works on OSes that are not Linux. I'd be interested to know which ones you are talking about. I have not heard of people using sbuild outside of Linux.

Then you state that debspawn is faster due to zstd tarballs and eatmydata. sbuild supports that as well plus lz4 tarballs which are again a tiny bit faster. Can you provide benchmarks to back up your claim?

Thanks!

cheers, josch

Transferred from BTS https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024091

The builder user is created with /nonexistent as the home directory, and
somewhere between the call to dsrun build-prepare and dsrun build-run
one of the calls to systemd-nspawn -u builder creates the home directory
of the user.

This is bad because this disables the security effect of having an
account's home directory point to a nonexistent directory, and also
because adduser's autopkgtests fail in such an environment since
adduser's autopkgtests test whether the special-case of /nonexistent
(which adduser takes care to never create) is handled correctly.

just like what sbuild does -- so pasting build results around would be easier.

debspawn fails to sign a source build produced with --only source. After build finished, debsign complains about missing amd64.changes file, it's expected such file to be missing in a source only build (as those required by Launchpad).

This patch works for me:

diff --git a/debspawn/build.py b/debspawn/build.py
index 974aa76..1dcb6a3 100644
--- a/debspawn/build.py
+++ b/debspawn/build.py
@@ -517,7 +517,8 @@ def build_from_directory(
 
     # sign the resulting package
     if sign:
-        r = _sign_result(osbase.results_dir, pkg_sourcename, pkg_version, osbase.arch)
+        sign_arch = 'source' if build_only == 'source' else osbase.arch
+        r = _sign_result(osbase.results_dir, pkg_sourcename, pkg_version, sign_arch)
         if not r:
             return False
 

a proper PR will need some refactor because _sign_result is called in another place.

This is with 0.4.2-2

$ debspawn create impish
[ debootstrap ]
I: Base system installed successfully.

┌───────────────────┐
│  Configure        │
└───────────────────┘
Ign:1 http://archive.ubuntu.com/ubuntu impish InRelease
Ign:1 http://archive.ubuntu.com/ubuntu impish InRelease
Ign:1 http://archive.ubuntu.com/ubuntu impish InRelease
Err:1 http://archive.ubuntu.com/ubuntu impish InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists...
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/impish/InRelease  Temporary failure resolving 'archive.ubuntu.com'
E: Some index files failed to download. They have been ignored, or old ones used instead.
Command `apt-get -uyq -o Dpkg::Options::="--force-confnew" update` failed.

I guess this is something to do with it, no idea atm why this happens. After a manual debootstrap:

ubuntu@rousing-razorfish:~$ ls impish/etc/resolv.conf -l
lrwxrwxrwx 1 root root 39 Jun  1 18:27 impish/etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
ubuntu@rousing-razorfish:~$ ls unstable/etc/resolv.conf -l
-rw-r--r-- 1 root root 928 Jun  1 18:26 unstable/etc/resolv.conf

and unstable works great. The symlink is broken inside the impish chroot of course, because we don't -boot it to actually run resolved there.

if I remove that resolv.conf then nspawn can work it out fine, so maybe one thing debspawn could do is remove it.

One of the things that tend to take the longest (especially for small packages on foreign arches) when building packages with Debspawn for me is waiting for the container to be prepared every time. In particular, hitting the following line in Installing package build-dependencies:

Setting up man-db (2.8.3-2ubuntu0.1) ...
Building database of manual pages ...

While it's usually not a big deal, it also seems extremely unnecessary for an ephemeral container to build manpages. I think this could be done by default, and you probably don't need any flags for it unless someone complains either, it should pretty much be a free speedup. If you agree, this patch ought to add that functionality:

diff --git a/debspawn/osbase.py b/debspawn/osbase.py
index 3df261d..6612763 100644
--- a/debspawn/osbase.py
+++ b/debspawn/osbase.py
@@ -614,6 +614,20 @@ class OSBase:
                             '--keep-baud - 115200,38400,9600 $TERM\n'
                         )
                     )
+            # prevent manpages from being built in the container
+            dpkg_conf_dir = Path(tdir, 'etc/dpkg/dpkg.conf.d')
+            os.makedirs(dpkg_conf_dir, exist_ok=True)
+            with open(dpkg_conf_dir / '01_nodoc', 'w') as f:
+                f.write(
+                    '''\
+path-exclude /usr/share/doc/*
+# we need to keep copyright files for legal reasons
+path-include /usr/share/doc/*/copyright
+path-exclude /usr/share/man/*
+path-exclude /usr/share/groff/*
+path-exclude /usr/share/info/*
+'''
+                )
 
             # delete unwanted files, especially resolv.conf as a broken one will
             # mess with the next step

Though of course, that doesn't include any of the actual things needed for a release. If you want, I'd be happy to put in a PR for it, but I figure it's probably better to leave that to you, so I don't mess up any conventions for the git log or whatever.

Hello, in the osbase run function it looks like the init command is only ever used if self._cachekey is truthy, and if the cached image doesn't already exist.

I noticed this when I tried to use --init-command, and I couldn't see any effect:

┌─────────────────────────────┐
│  Preparing container        │
└─────────────────────────────┘
Hit:1 http://deb.debian.org/debian sid InRelease
Hit:2 http://deb.debian.org/debian experimental InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

┌──────────────────────┐
│  Running Task        │
└──────────────────────┘
regular command

┌────────────────────────────────────┐
│  Retrieving build artifacts        │
└────────────────────────────────────┘
Copied 0 files.
Done

With a cachekey, the init command is run as expected:

╔════════════════════════════════════════════════╗
║  Preparing template for `test_cachekey`        ║
╚════════════════════════════════════════════════╝

┌─────────────────────────────┐
│  Preparing container        │
└─────────────────────────────┘
Hit:1 http://deb.debian.org/debian sid InRelease
Hit:2 http://deb.debian.org/debian experimental InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
/srv
uid=0(root) gid=0(root) groups=0(root)
init command
Storing prepared image in cache
New compressed tarball size (for test_cachekey) is 610.0MiB
Using cached container image `test_cachekey`
usermod: no changes

┌─────────────────────────────┐
│  Preparing container        │
└─────────────────────────────┘
Hit:1 http://deb.debian.org/debian sid InRelease
Hit:2 http://deb.debian.org/debian experimental InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

┌──────────────────────┐
│  Running Task        │
└──────────────────────┘
regular command

┌────────────────────────────────────┐
│  Retrieving build artifacts        │
└────────────────────────────────────┘
Copied 0 files.
Done.

The doc says something about the init command being ignored if the cachekey already exists (which is correct!), but not about the init command being ignored if there is no cachekey, so I assume this may be a bug?