The dnf stage currently takes a checksum of the repository metadata and the names of the packages that need to be installed, without their dependencies. This is to ensure that a pipeline is pinned to specific versions of packages. It effectively means that we cannot use repositories that change a lot, because if any package changes, the metadata checksum changes. We don't need to be that strict, though: a pipeline doesn't need to fail if any package in that repository changes, but only if a package that is needed for that pipeline changes.

To make this work, we have to move depsolving out of the stage: Put all packages that are needed for an image into the pipeline, including their href and pkid. We want a separate stage for it: org.osbuild.rpm. It would use rpm to install all packages and and verify their signatures.

Notes:

• We can verify/check an RPM's pkgid by examining the SHA256 item in the signature header - rpm --qf '%{SHA256HEADER}\n' -qp $RPM - or just do a SHA256 of the package's header section ourselves. We can also ask RPM to verify the a package's signatures/digests with rpm -K.

• DNF metadata isn't signed, only the packages themselves. So yeah, if we've previously verified the package signatures for a given pkgid, then we can skip the GPG check inside the pipeline.

• If we want/need stronger guarantees about package/signature validity inside the pipeline, it might make sense for us to construct a "sigpkgid" - either a hash of both RPM header sections, or just a hash for the signature section - which would let us uniquely identify a package and its signatures, and thus be sure that we're only using previously-validated packages.

Caching would be great to have, but is out of scope for this. There are just too many open questions around it now, especially how it would fit into osbuild's current share-nothing model.

There is a substantial difference between the simple MBR image and using gpt: 9s vs 18s. I wonder what is going on.

Each repository passed to org.osbuild.dnf can have a gpgkey specified. Right now, this is defined as the fingerprint of the key. The stage then fetches it from whatever keyserver is configured in the buildroot. The keyservers don't seem to be reliable, though, leading to lots of unnecessary test failures.

The fingerprint is semantically equivalent to the public key. Let's just include the public key in the pipeline, as the text produced by gpg2's --export --armor options.

All examples and test pipelines in osbuild need to be updated, and all code that uses it in osbuild-composer. osbuild is not stable yet, so a change like this is fine.

The temporary tree is moved to the ObjectStore.objects folder via rename(3) (os.rename). If the target already exists (and is non-empty) this will lead to either ENOTEMPTY or EEIXST. The latter is currently not ignored, which leads at least on btrfs to:

 File "/home/gicmo/Code/src/osbuild/osbuild/objectstore.py",
     os.rename(tree.root, output_tree)
 FileExistsError: [Errno 17] File exists: 'store/tmpyyi3yvie/tree' -> 'store/objects/…'

I've recently stumbled over this project thanks to @Conan-Kudo and it has quite the overlap with KIWI: https://github.com/OSInside/kiwi/, which is a operating system image builder.

KIWI has support for a lot of different image types, from installation ISOs, over netboot images and VMs, to OCI containers. It is mostly used to build the images for the openSUSE und SLE operating systems in the open build service (https://build.opensuse.org), but it is also available on Fedora and CentOS.

I was wondering if you would be interested in collaborating with KIWI? KIWI is written in Python and exposes most of its functionality via a Python API, so you'd be able to reuse parts of it.

CC the main KIWI maintainers: @schaefi @davidcassany @Conan-Kudo

It should be possible to use an rpmostree stage in place of the dnf/rpm stages we currently use in all our pipelines.

This is a place-holder issue so people can keep track of this for when/if we start working on it.

Running this pipeline:

{
  "build": {
    "pipeline": {
      "stages": [
        {
          "name": "org.osbuild.error"
        }
      ]
    },
    "runner": "org.osbuild.fedora31"
  },
  "stages": [
    {
      "name": "org.osbuild.noop"
    }
  ],
  "assembler": {
    "name": "org.osbuild.noop"
  }
}

results osbuild crashing:

Traceback (most recent call last):
  File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/lars/src/osbuild/osbuild/__main__.py", line 121, in <module>
    sys.exit(main())
  File "/home/lars/src/osbuild/osbuild/__main__.py", line 99, in main
    secrets=secrets
  File "/home/lars/src/osbuild/osbuild/pipeline.py", line 356, in run
    secrets)
  File "/home/lars/src/osbuild/osbuild/pipeline.py", line 238, in build_stages
    t.cleanup()
AttributeError: 'NoneType' object has no attribute 'cleanup'

This patch works for me, but I'm not sure if there's not something else going on:

@@ -234,11 +234,13 @@ class Pipeline:
                                             secrets)
             # Cleanup the build tree used to build `tree`
             # which is not needed anymore
-            t.cleanup()
+            if t:
+                t.cleanup()
 
             results["build"] = r
             if not r["success"]:
-                tree.cleanup()
+                if tree:
+                    tree.cleanup()
                 results["success"] = False
                 return results, None, None

In this issue coreos/coreos-assembler#91 (comment) osbuild and coreos-assembler came up.

I looked over this project quickly; your Python code looks nice overall. But that said I don't see what could be shared between it and cosa offhand. cosa already solves problems like host independence by coming as a container. We avoid affecting (or depending on) the host by using supermin (from the libguestfs project) to run disposable VMs to do things like create filesystems.

Or to say it another way, I think http://libguestfs.org/ is a great project and we're already reusing it. Our bits to generate bootable disk images are hence just a small shell script.

Obviously this project also wasn't created with OSTree in mind, whereas coreos-assembler is entirely designed around rpm-ostree.

That also leads to some other important things; for example the versioning.

Anyways, happy to discuss things here!

Error:

usermod: failure while writing changes to /etc/shadow
Traceback (most recent call last):
  File "/run/osbuild/lib/stages/org.osbuild.users", line 103, in <module>
    r = main(args["tree"], args["options"])
  File "/run/osbuild/lib/stages/org.osbuild.users", line 90, in main
    usermod(tree, name, gid, groups, description, home, shell, password)
  File "/run/osbuild/lib/stages/org.osbuild.users", line 55, in usermod
    subprocess.run(["chroot", root, "usermod", *arguments, name], check=True)
  File "/usr/lib64/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['chroot', '/run/osbuild/tree', 'usermod', '--password', '$6$YjuXsenagq97bEyX$vHpP1.KwH7t1hjchYM0EhM54q5vLrNcz4/jj874gKGH.J.F.ShKt9QK/9iH24z1n1iwCt0WkZNxVmORbXHZG3/\n', 'test']' returned non-zero exit status 1.

I suspect selinux is reponsible for this, in the container after first user's passowrd is set the output of ls -lZ /etc/shadow is following:

----------. 1 root root system_u:object_r:etc_t:s0 1411  Sep 5 09.22 /etc/shadow

On my laptop's Fedora installation, the output is following:

----------. 1 root root system_u:object_r:shadow_t:s0 1411  Sep 3 12.15 /etc/shadow

I guess we need to change the tag?

Sample for reproduce: (first password is password, second password is password2)

{
  "name": "base-qcow2",
  "stages": [
    {
      "name": "org.osbuild.dnf",
      "options": {
        "releasever": "30",
        "install_weak_deps": true,
        "repos": {
          "fedora": {
            "name": "Fedora",
            "metalink": "https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch",
            "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch"
          }
        },
        "packages": [
          "@Core",
          "chrony",
          "kernel",
          "selinux-policy-targeted",
          "grub2-pc",
          "spice-vdagent",
          "qemu-guest-agent",
          "xen-libs",
          "langpacks-en"
        ]
      }
    },
    {
      "name": "org.osbuild.locale",
      "options": {
        "language": "en_US"
      }
    },
    {
      "name": "org.osbuild.fstab",
      "options": {
        "filesystems": [
          {
            "uuid": "76a22bf4-f153-4541-b6c7-0332c0dfaeac",
            "vfs_type": "ext4",
            "path": "/",
            "freq": "1",
            "passno": "1"
          }
        ]
      }
    },
    {
      "name": "org.osbuild.grub2",
      "options": {
        "root_fs_uuid": "76a22bf4-f153-4541-b6c7-0332c0dfaeac",
        "kernel_opts": "ro biosdevname=0 net.ifnames=0"
      }
    },
    {
      "name": "org.osbuild.selinux",
      "options": {
        "file_contexts": "etc/selinux/targeted/contexts/files/file_contexts"
      }
    },
    {
      "name": "org.osbuild.fix-bls"
    },
    {
      "name": "org.osbuild.users",
      "options": {
        "users": {
          "test": {
            "password": "$6$s06sY2/bZbmuyE4a$JTn/Ki2jbNG.wGeCvt6mcjaIZ2HV.H8VjLzIMZs7f2G48NMFlcB2/OhbHZ6HhZAOcb.CmUb2qywSW2DkKe6gn0"
          }
        }
      }
    },
    {
      "name": "org.osbuild.users",
      "options": {
        "users": {
          "test": {
            "password": "$6$YjuXsenagq97bEyX$vHpP1.KwH7t1hjchYM0EhM54q5vLrNcz4/jj874gKGH.J.F.ShKt9QK/9iH24z1n1iwCt0WkZNxVmORbXHZG3/\n"
          }
        }
      }
    }
  ],
  "assembler":
    {
      "name": "org.osbuild.qcow2",
      "options": {
        "filename": "base.qcow2",
        "root_fs_uuid": "76a22bf4-f153-4541-b6c7-0332c0dfaeac",
        "size": 3221225472
      }
    }
}

Give each stage and assembler a machine and human readable documentation.

Assemblers:

• org.osbuild.noop
• org.osbuild.tar
• org.osbuild.qemu
• org.osbuild.rawfs

Stages:

• org.osbuild.noop
• org.osbuild.chrony
• org.osbuild.groups
• org.osbuild.hostname
• org.osbuild.locale
• org.osbuild.selinux
• org.osbuild.firewall
• org.osbuild.grub2
• org.osbuild.keymap
• org.osbuild.systemd
• org.osbuild.timezone
• org.osbuild.fstab
• org.osbuild.users
• org.osbuild.dnf
• org.osbuild.yum
• org.osbuild.rpm
• org.osbuild.fix-bls
• org.osbuild.test
• org.osbuild.debug-shell
• org.osbuild.script

As mentioned in #161, some lorax-composer image types perform high-level tweaks to network configuration. These are:

• Disable device renaming (ami, hyper-v, openstack, vhd)
• Disable ipv6 (vhd, google)
• Set MTU (google)
• Disable firewall (google, openstack)

Example:

python3 -m osbuild --libdir . samples/f30-s390x.json 

org.osbuild.dnf: 5f16d42dca98e30d4e30c069795980e73078e3b7c9913bd6a75b040ac60d81f1 {
  "releasever": "30",
  "basearch": "s390x",
  "install_weak_deps": true,
  "repos"...
}

execv(/run/osbuild/lib/runners/org.osbuild.host) failed: No such file or directory

Failed

Fix the runner, re-run the pipeline:

# python3 -m osbuild --libdir . samples/f30-s390x.json 

Assembler org.osbuild.qemu: 4eab4c54ea4ef55c6bcd06de43e96e45768748854a5cfa872cd37aafa4f83e0a {
  "bootloader": "zipl",
  "format": "qcow2",
  "filename": "base.qcow2",
  "ptuuid": "0x14fc63d2",
  "root_fs_uuid": "76a22bf4-f153-4541-b6c7-0332c0dfaeac",
  "size": 3221225472
}

[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
[/usr/lib/tmpfiles.d/subscription-manager.conf:1] Line references path below legacy directory /var/run/, updating /var/run/rhsm → /run/rhsm; please update the tmpfiles.d/ drop-in file accordingly.
Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system
mke2fs 1.44.6 (5-Mar-2019)
Discarding device blocks: done                            
Creating filesystem with 786176 4k blocks and 196608 inodes
Filesystem UUID: 76a22bf4-f153-4541-b6c7-0332c0dfaeac
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

Traceback (most recent call last):
  File "/run/osbuild/lib/assemblers/org.osbuild.qemu", line 568, in <module>
    ret = main(args["tree"], args["output_dir"], args["options"], remoteloop.LoopClient(sock))
  File "/run/osbuild/lib/assemblers/org.osbuild.qemu", line 538, in main
    install_zipl(root, disk, pt)
  File "/run/osbuild/lib/assemblers/org.osbuild.qemu", line 472, in install_zipl
    kernel, initrd = find_kernel(root)
  File "/run/osbuild/lib/assemblers/org.osbuild.qemu", line 449, in find_kernel
    for dirent in os.scandir(base):
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/osbuild-mntxnxsdwu6/boot/loader/entries'

Failed

It seems like object store contains the tree_id (or output_id) even though the stage/assembler failed.

The result objects should contain the rpm metadata for each installed rpm. This is needed by composer to successfully submit a compose to koji, and the information is only from the rpms themselves (which composer does not have access to).

For s390x (added with #189) we discover the available kernels and their options and configuration in order to feed zipl (the z Initial Program Loader) the right parameters. The Boot Loader Specification itself does in the current revision explicitly omit to specify what entry should be the default to boot, but @martinezjavier explained to me that grub2 and zipl use "the rpm sorting algorithm: rpmvercmp()". Therefore we should do the same to sort the entries and pick the latest one as the default one.

On those of our images that require a bootloader, we should have tests to verify that it has been installed correctly, in the same way we verify the partition table and the filesystem. We currently only support GRUB2, so the test should be specific to that.

I am currently not aware of any convenient tool to inspect the installed bootloader, so unless someone knows of a better way we should read the bytes from the raw block device, hash them and verify them against an expected value.

GRUB2 comes in two stages, the first one is shipped as a static blob in the grub2-pc package and it is the first 440 bytes of the image. The second stage is generated and can in principle stretch until the start of the first partition (maybe verify if grub2 makes some guarantee about its max size), starting at a 512 byte offset from the beginning of the device. I suggest hashing these two areas independently and including the hashes when verifying our images.

There are some places where the assumption that /boot is on /, i.e. the root partition, is hard-coded:

• org.osbuild.grub2:
  • grubenv
  • redirect grub.cfg
• org.osbuild.fix-bls

We need a way to test that osbuild installed from the RPM package works on a fresh installation of Fedora. We used to have the Vagrant test, but that's gone so we need a new way. If we had something like this it could have prevented errors like #145 .

After killing osbuild with SIGINT (ctrl-c) several mounts are not cleaned up:

$ findmnt
TARGET                                          SOURCE                                   FSTYPE     OPTIONS
/                                               /dev/vda1                                ext4       rw,relatime,seclabel
├─/run                                          tmpfs                                    tmpfs      rw,nosuid,nodev,seclabel,mode=755
│ ├─/run/osbuild/osbuild-buildroot-yw6zbs5h/usr /dev/vda1[/usr]                          ext4       ro,relatime,seclabel
│ └─/run/osbuild/osbuild-buildroot-yw6zbs5h/var /dev/vda1[/var/tmp/osbuild-var-0jquja48] ext4       rw,relatime,seclabel
├─/var/tmp/tmp9mo305je/tmpwcwdxgpa              /dev/vda1                                ext4       ro,relatime,seclabel

We now support combined Sources and Pipeline input to the osbuild executable. Before this, we took the pipeline description directly. For compatibility reasons, we still support this old way of passing the input. Make sure we rip it out once the next release is out.

As first part of #238 we need to investigate how ostree fits into the osbuild pipeline model, i.e. how to create ostree trees and fetch ostree trees. As a first proof-of-concept use an existing tree, fetch it and assemble it with the qemu assembler.

Currently the org.osbuild.grub2 stage uses a grubenv which is mainly used for two things:

• setting the file-system uuid for the "grub" filesystem
• kernel options

There are some plans to not use $kernelopts and instead modified bls snippets directly when there is a need to change kernel options. cc @martinezjavier

Currently the trees that are created for the build pipelines are always committed to the ObjectStore. The reason for this is "merely" an implementation detail but not a design decision. There is not reason why any tree should be committed automatically. The tricky part for build pipeline commits is the automatic lifetime management of the objectstore.Objects in combination with the fact that the Assembler also needs the build tree.

osbuild caches trees, but not output from assemblers. This is inconsistent, because trees are cached, but outputs are not. People (or programs like Composer) that use osbuild need to manage where they store the outputs themselves, using --output. That argument is a bit awkward to use, because one (technically) needs to clear it between every run of osbuild, or pass a new one named after the pipeline.

osbuild can manage this for them by storing all output in .osbuild/output, which contains directories named after the pipline's hash, similar to .osbuild/objects.

This lets us remove the --output argument. To make the last output easier to find, let's print the full path after the assembler is finished running.

I tried to run 6 osbuilds parallelly. Sometimes, dnf command inside dnf stages failed. The crashes were due to various reasons. I've attached couple of logs from failed runs.

The test environment:

• Fedora 30 in Vagrant/libvirt VM on Fedora 30 host
• ok-ish internet connection (router is kinda old, it may be responsible for dropping packets)

Logs:

• https://gist.githubusercontent.com/ondrejbudai/447dc0f4400f954f9193d79bb3d1b70f/raw/15b4bf2cee81d426cb5b513d9407e1ae1f421d67/dnf-fail-1
• https://gist.githubusercontent.com/ondrejbudai/699aa3466e8c555acd2b9716a30608cf/raw/f831447faa68e9d1ed1a16ae70fe607b8da49a62/dnf-fail-2

We want to be able to introspect both the build tree and the namespaces of the container in case of failure. Ideally without relying on tools from within either.

See PR #60.

When installing a package that triggers dracut to be run, we get a lot of output like this:

/usr/lib/dracut/dracut-logger.sh: line 341: /dev/kmsg: Operation not permitted

Dracut falls back to logging to kmsg because /dev/log doesn't exist (see dracut-logger.sh:156).

As a workaround, I've bind-mounted /dev/kmsg when debugging.

This is probably a bug in dracut. I think it should fall back to logging to stderr only when there's nothing else to log to. However, it would be nice to have a workaround for now.

Now that we no longer do any implicit checkpointing, we should make it tolerably easy to make explicit ones. I think the simplest solution would be to have some sort of --info mode where you can parse a pipeline and get information about it, such as its tree ids. Then callers could first extract these, and then checkpoint the ones we want.

The typical use-cases I see are to checkpoint the build tree, the result of the dnf/rpm stage, and for rpmostree support: the final stage.

PR #282 provides a first implementation of ostree support for osbuild. There are a couple of outstanding issues:

Essential:

• stage/assembler descriptions
• pylint issues

Required:

• SELinux fixups
• set upstream ref to enable updates
• more testing, including updating the image

Optionally, if there is time otherwise later:

• parent commit support
• merge grub stage
• correct metadata for ostree commit

I just got the following error when trying to build qcow2 image in cockpit-composer:

pro 09 09:01:40 vm systemd[1]: Started Container osbuild-buildroot-s84c2zdz.
pro 09 09:01:41 vm kernel: loop_set_status: loop0 () has still dirty pages (nrpages=1)
pro 09 09:01:41 vm osbuild-worker[24503]: Exception in callback LoopServer._dispatch()
pro 09 09:01:41 vm osbuild-worker[24503]: handle: <Handle LoopServer._dispatch()>
pro 09 09:01:41 vm osbuild-worker[24503]: Traceback (most recent call last):
pro 09 09:01:41 vm osbuild-worker[24503]:   File "/usr/lib64/python3.6/asyncio/events.py", line 145, in _run
pro 09 09:01:41 vm osbuild-worker[24503]:     self._callback(*self._args)
pro 09 09:01:41 vm osbuild-worker[24503]:   File "/usr/lib/python3.6/site-packages/osbuild/remoteloop.py", line 95, in>
pro 09 09:01:41 vm osbuild-worker[24503]:     devname = self._create_device(fd, dir_fd, offset, sizelimit)
pro 09 09:01:41 vm osbuild-worker[24503]:   File "/usr/lib/python3.6/site-packages/osbuild/remoteloop.py", line 80, in>
pro 09 09:01:41 vm osbuild-worker[24503]:     lo.set_status(offset=offset, sizelimit=sizelimit, autoclear=True)
pro 09 09:01:41 vm osbuild-worker[24503]:   File "/usr/lib/python3.6/site-packages/osbuild/loop.py", line 208, in set_>
pro 09 09:01:41 vm osbuild-worker[24503]:     fcntl.ioctl(self.fd, self.LOOP_SET_STATUS64, info)
pro 09 09:01:41 vm osbuild-worker[24503]: BlockingIOError: [Errno 11] Resource temporarily unavailable

The bad thing about this is that osbuild-worker just freezed and I needed to restart it manually.

Current, we support only the fs_spec with UUID, see the following example:

Stage options:

"options": {
  "filesystems": [
    {
      "uuid": "76a22bf4-f153-4541-b6c7-0332c0dfaeac",
      "vfs_type": "ext4",
      "path": "/",
      "freq": "1",
      "passno": "1"
    }
  ]
}

Result in /etc/fstab:

UUID=76a22bf4-f153-4541-b6c7-0332c0dfaeac / ext4 defaults 1 1

With this approach we are not able to mount some filesystems, e.g. tmpfs:

tmpfs /tmp tmpfs rw,nodev,nosuid,size=2G 0 0

I don't know whether we need it. I'm also not sure about the format of tmpfs stage options. Currently it has the option uuid. I think the cleanest approach is to drop the uuid option and introduce a new option (named spec possibly? see /etc/fstab). The difference between uuid and spec options is that in spec option user must explicitly prepend "UUID=" string if uuid spec is needed.

osbuild fails when the build pipeline doesn't have a runner set. It should probably use the host runner in that case, similarly to when there's no build pipeline at all.

{
  "build": {
    "pipeline": {
      "stages": [
        {
          "name": "org.osbuild.noop"
        }
      ]
    }
  },
  "assembler": {
    "name": "org.osbuild.noop"
  }
}

Traceback (most recent call last):
  File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/lars/src/osbuild/osbuild/__main__.py", line 121, in <module>
    sys.exit(main())
  File "/home/lars/src/osbuild/osbuild/__main__.py", line 74, in main
    pipeline = osbuild.load(pipeline, sources_options)
  File "/home/lars/src/osbuild/osbuild/pipeline.py", line 414, in load
    build_pipeline, runner = load_build(build, sources_options)
  File "/home/lars/src/osbuild/osbuild/pipeline.py", line 408, in load_build
    return build_pipeline, description["runner"]
KeyError: 'runner'

For the kickstarts used by lorax-composer for the target images in osbuild-composer (primarily targeting RHEL8.2 and secondarily Fedora 31), document any configuration quirks that are performed on top of what we currently do. Document each as a separate issue, assigning it to yourself for the the osbuild 7 milestone. Then implement the stages necessary for realising these tweaks using osbuild.

The results returned to the caller when building a pipeline should be the same whether or not parts of the pipeline was cached in the store. This will be important when we add metadata to the results object that the caller can depend upon. In particular, the org.osbulid.rpm stage will return rpm metadata, which composer will pass on to koji.
Suggested implementation: store the tree as a subdirectory of the object in the object store, and the result of the pipeline up to that point as a json file. When reusing a previous tree, also prepend the associated result object.

Requirements:

• local checkout should enable make srpm where the version is extracted from git
• packit should be able to generate the version automatically for CI builds
• packit should correctly substitute versions when proposing new PRs to Pagure
• Github releases should contain specific version because they don't contain git metadata

I got stuck because the last requirement seems to be impossible to fulfill. The first 3 requirements can be met using setup.py sdist (see Makefile for example), but the Github releases are always simple git archive which basically require us to have a file with a version written inside. Any ideas? Not use Github releases? Upload different files to them? Extract the version from directory name (github tarballs are named osbuild-version, so in theory the version is in the name)?

Usages of runner should keep in mind that it is a nullable variable:

Result:

~/osbuild-composer> cat test/cases/vhd_local_boot.json | jq .pipeline | sudo osbuild --store /var/osbuild -

org.osbuild.dnf: 2f23447c3243d936cce909d9f2d943c52df07b95271f339f920df3aff8bda5df {
  "repos": [
    {
      "metalink": "https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch",
      "gpgkey": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBFturGcBEACv0xBo91V2n0uEC2vh69ywCiSyvUgN/AQH8EZpCVtM7NyjKgKm\nbbY4G3R0M3ir1xXmvUDvK0493/qOiFrjkplvzXFTGpPTi0ypqGgxc5d0ohRA1M75\nL+0AIlXoOgHQ358/c4uO8X0JAA1NYxCkAW1KSJgFJ3RjukrfqSHWthS1d4o8fhHy\nKJKEnirE5hHqB50dafXrBfgZdaOs3C6ppRIePFe2o4vUEapMTCHFw0woQR8Ah4/R\nn7Z9G9Ln+0Cinmy0nbIDiZJ+pgLAXCOWBfDUzcOjDGKvcpoZharA07c0q1/5ojzO\n4F0Fh4g/BUmtrASwHfcIbjHyCSr1j/3Iz883iy07gJY5Yhiuaqmp0o0f9fgHkG53\n2xCU1owmACqaIBNQMukvXRDtB2GJMuKa/asTZDP6R5re+iXs7+s9ohcRRAKGyAyc\nYKIQKcaA+6M8T7/G+TPHZX6HJWqJJiYB+EC2ERblpvq9TPlLguEWcmvjbVc31nyq\nSDoO3ncFWKFmVsbQPTbP+pKUmlLfJwtb5XqxNR5GEXSwVv4I7IqBmJz1MmRafnBZ\ng0FJUtH668GnldO20XbnSVBr820F5SISMXVwCXDXEvGwwiB8Lt8PvqzXnGIFDAu3\nDlQI5sxSqpPVWSyw08ppKT2Tpmy8adiBotLfaCFl2VTHwOae48X2dMPBvQARAQAB\ntDFGZWRvcmEgKDMwKSA8ZmVkb3JhLTMwLXByaW1hcnlAZmVkb3JhcHJvamVjdC5v\ncmc+iQI4BBMBAgAiBQJbbqxnAhsPBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAK\nCRDvPBEfz8ZZudTnD/9170LL3nyTVUCFmBjT9wZ4gYnpwtKVPa/pKnxbbS+Bmmac\ng9TrT9pZbqOHrNJLiZ3Zx1Hp+8uxr3Lo6kbYwImLhkOEDrf4aP17HfQ6VYFbQZI8\nf79OFxWJ7si9+3gfzeh9UYFEqOQfzIjLWFyfnas0OnV/P+RMQ1Zr+vPRqO7AR2va\nN9wg+Xl7157dhXPCGYnGMNSoxCbpRs0JNlzvJMuAea5nTTznRaJZtK/xKsqLn51D\nK07k9MHVFXakOH8QtMCUglbwfTfIpO5YRq5imxlWbqsYWVQy1WGJFyW6hWC0+RcJ\nOx5zGtOfi4/dN+xJ+ibnbyvy/il7Qm+vyFhCYqIPyS5m2UVJUuao3eApE38k78/o\n8aQOTnFQZ+U1Sw+6woFTxjqRQBXlQm2+7Bt3bqGATg4sXXWPbmwdL87Ic+mxn/ml\nSMfQux/5k6iAu1kQhwkO2YJn9eII6HIPkW+2m5N1JsUyJQe4cbtZE5Yh3TRA0dm7\n+zoBRfCXkOW4krchbgww/ptVmzMMP7GINJdROrJnsGl5FVeid9qHzV7aZycWSma7\nCxBYB1J8HCbty5NjtD6XMYRrMLxXugvX6Q4NPPH+2NKjzX4SIDejS6JjgrP3KA3O\npMuo7ZHMfveBngv8yP+ZD/1sS6l+dfExvdaJdOdgFCnp4p3gPbw5+Lv70HrMjA==\n=BfZ/\n-----END PGP PUBLIC KEY BLOCK-----\n",
      "checksum": "sha256:9f596e18f585bee30ac41c11fb11a83ed6b11d5b341c1cb56ca4015d7717cb97"
    }
  ],
  "packages": [
    "dnf",
    "e2fsprogs",
    "policycoreutils",
    "qemu-img",
    "systemd",
    "grub2-pc",
    "tar"
  ],
  "releasever": "30",
  "basearch": "x86_64"
}

execv(/run/osbuild/lib/runners/None) failed: No such file or directory

org.osbuild.dnf failed with code 1

Pipeline:
https://github.com/osbuild/osbuild-composer/blob/master/test/cases/vhd_local_boot.json

Some platforms (notably Azure) require that the initramfs contains certain binaries, modules, or kernel drivers in order to boot properly.

This means osbuild needs a way to modify the contents of the image's current initramfs (if it has one), and to make those configuration changes persistent so any future initramfs built inside the image (e.g. on a future kernel update) will also contain the needed pieces.

lorax-composer and other tools handle this by creating files in /etc/dracut.conf.d/ in kickstart %post scriptlets - see vhd.ks for an example.

The best way for osbuild to handle this is probably to have a stage that allows creating dracut config files and building new initrd images for any/all kernel(s) present in the tree.

For feature parity with lorax-composer and current official RHEL images, this stage needs to support at least the following dracut.conf(5) options:

• add_drivers
• add_dracutmodules
• install_items
• dracut_rescue_image (most image types remove the rescue image to save space)
• early_microcode (on by default but some image types turn it off)
• omit_dracutmodules (e.g. live-img removes plymouth)

After modifying the dracut config, the initrd images have to be (re)generated by (re)running dracut. See pylorax.treebuilder.rebuild_initrds() and pylorax.creator.rebuild_initrds_for_live() for details on how lorax-composer handles finding the correct kernel version, output filename, etc.

I think this pipeline should fail, because the build pipeline fails:

{
  "build": {
    "pipeline": {
      "stages": [
        {
          "name": "org.osbuild.error"
        }
      ]
    },
    "runner": "org.osbuild.fedora31"
  },
  "assembler": {
    "name": "org.osbuild.noop"
  }
}

However, it works. The assembler is run on the host. Added a single stage makes it fail while building the build pipeline.

Many users who build images need to be able to tweak ssh/sshd configuration. Some platforms (e.g. Microsoft Azure, Google Compute Engine) expect certain ssh/sshd settings, and in our own testing we've occasionally had to change things like which port sshd listens on.

osbuild should allow users to change (or replace) the image's ssh_config or sshd_config, so the image never has to come online with an "unsafe" or "unsupported" configuration.

Kickstart doesn't offer a command to configure sshd, so kickstart users handle this in %post scriptlets, like lorax-composer's google.ks. Ansible uses generic file-management modules for this (see examples using template, replace, blockinfile, assemble).

What needs to be configured?

Google's documentation on configuring custom images has the following recommendations:

• Disable root ssh login.
• Disable password authentication.
• Disable host-based authentication.
• Enable strict, host-key checking.
• Use ServerAliveInterval to keep connections open.

Microsoft's documentation on how to prepare a Red Hat-based virtual machine for Azure directs the user to

Modify /etc/ssh/sshd_config to include the following line:

ClientAliveInterval 180

So, to be compatible with lorax-composer and information provided by Amazon, Google and Microsoft osbuild should be able to specify at least the following sshd_config(5) settings:

• PasswordAuthentication
• PermitRootLogin
• PermitEmptyPasswords
• ClientAliveInterval
• PermitTunnel
• AllowTcpForwarding
• X11Forwarding
• LoginGraceTime
• SyslogFacility
• Ciphers
• MACs
• Port
• ListenAddress

Google also has recommended settings for these ssh_config(5) items:

• Host
• Protocol
• ForwardAgent
• ForwardX11
• HostbasedAuthentication
• StrictHostKeyChecking
• Ciphers
• Tunnel
• ServerAliveInterval

Running this command from the test suite:

sudo qemu-system-x86_64 -nographic -m 1024 -snapshot -accel kvm:hvf:tcg /tmp/osbuild-7nc58zbl/web-server.qcow2 -net nic,model=rtl8139 -net user,hostfwd=tcp::8888-:8888

results in the following error:

error: ../../grub-core/fs/fshelp.c:257:file
`/osbuild-px27al9e/tmp6_9916qr/tree/boot/vml
inuz-5.0.9-301.fc30.x86_64' not found.
error: ../../grub-core/loader/i386/pc/linux.c:413:you need to load the kernel
first.

Press any key to continue...

  Failed to boot both default and fallback entries.

Press any key to continue...

This has landed in the latest nightlies: https://bugzilla.redhat.com/show_bug.cgi?id=1766754

I wonder when it's ok for us to drop hacks. Since RHEL 8.2 is not yet out, I think it should be fine? On the other hand, people might be using pre releases that don't have the fix yet.

The current experimental ostree implementation (see #282) does not yet support ignition. Investigate what are the necessary to enable it in the image as well as a proper testing environment for it.

TODO:

• ignition packages
• dracut ignition module
• partition / filesystem need to be identified via label not uuid

tl;dr: osbuild needs to be able to configure a variety of different services (dracut, sshd, chrony, cloud-init, waagent, network-scripts, udev, ...) that use simple text configuration files. Do we want do handle that kickstart-style (a stage for each service), ansible-style (generic file-manipulation stages), or take a hybrid approach (make a tool/module that does safe generic file tweaks and use that to implement per-service stages)?

Details: As outlined in #161, for parity with lorax-composer we need to be able to configure the following things:

• dracut (see #180)
• sshd (see #181)
• WALinuxAgent (see vhd.ks)
• cloud-init (see vhd.ks, ami.ks, RCM's rhel-8*-ec2.ks)
• chrony (via kickstart timezone --nameservers - see google.ks)

These are five examples of miscellaneous high-level configuration tweaks where the actual implementation is basically just "set the value for this key in this {json,yaml,ini,key=val...}-format config file".

In kickstart-based image builds, anything that doesn't already have a kickstart command is handled in a %post scriptlet, using sed, cat << EOF, etc. Ansible does not ship modules for any of these tasks, but instead has generic modules for writing/modifying files: template, replace, template, lineinfile, blockinfile, ini_file, etc.

In an initial discussion last week, I preferred the generic approach, because years of dealing with user requests around kickstart commands and %post scripts make me think it's a bad idea to require osbuild developers to write a new stage for each new service or option our users need. It's too inflexible for users that have uncommon or proprietary services/tools, and those who are using osbuild but need something we don't provide will be stuck until we build and push a new osbuild release.

@larskarlitski and @teg were against the idea of generic stages, which is a sound principle, because the other big problem with Kickstart is that the "generic configuration" option (%post scripts) is far too generic and lets users make all kinds of wild and unintended changes that can completely break our tools or their systems in hard-to-debug ways. So we definitely need something that's safer than shell scripts.

It's also true that most of the things currently being handled in kickstart %post or Ansible playbooks don't need to be done during the image build. But some do, and we need some way to handle those things. So: thoughts?

We were early adopters of Packit few months ago, but we decided to disable it because it was spamming our PRs and didn't work most of the time. Before we reintroduce it, do a review of current state of the packit-as-a-service and check if it can:

• Automatically create PR in src.fedoraproject.org when we create a new release in Github
• Automatically generate a changelog from the upstream release notes

We don't need RPM build on push any more because we have our own solution in AWS.

Furthermore:

• It must not write a comment on each PR force-push
• It should not write any comments into the PRs
• It must understand our spec file and be able to process it (the same for osbuild-composer, because I don't have to have different workflow for each RPM)
• It must be more stable than the last time we used it (since we only want automatic releases and there is no place to write logs, it either has to work all the time or have the dashboard where we can see status)

We are at a point where osbuild is about to be actively deployed and used. This means there are systems that depend on it installing, updating, and running in a reliable fashion. Furthermore, it means there will be people stumbling over it, be it administrators, users, or just a passerby. We therefore should take care that the project has suitable documentation, landing-pages, a clear structure, and an agreed development style.

As part of this issue, I propose:

• Clean up all random files from the repository. Make sure everything has a purpose and clear structure.
• Clean up README.md, follow a common structure and point to relevant resources.
• Ensure downstream packagers can find required information at the common places. This means maintaining release notes (NEWS.md maybe?), having structured and predictable release paths, names, and procedures. This also includes licensing information, consistent copyright information (if any), install notes, yadda yadda yadda.
• Ensure there is a public landing-page with referrals to all our resources on https://www.osbuild.org.
• Provide a suitable documentation infrastructure. This requires reviewing runtime documentation (--help, etc), establishing install-time documentation (man-pages, etc.), and providing resources online including referrals to possible offline documentation.
• Agree on a backwards- and forwards-compatibility schema. How do we version things? How do we communicate breakage, and when are we allowed to commit breaks?
• Agree and document our development model. How do external contributors interact with us?

This is not an exhaustive list, yet. I will extend it, if anyone has suggestions.

I am on Fedora 31, using a fresh git clone (* master 27cf84e bind osbuild module from dynamically discovered path) and executing sudo strace -ff -o trace.log python3 -m osbuild --libdir . samples/base-qcow2.json I get:

org.osbuild.dnf: 550fa2834d028a59c74ce0a4400c1b6529ffbda27b30ba15fb14edc746b18695 {
  "releasever": "30",
  "basearch": "x86_64",
  "install_weak_deps": true,
  "repos": [
    {
      "metalink": "https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch",
      "checksum": "sha256:9f596e18f585bee30ac41c11fb11a83ed6b11d5b341c1cb56ca4015d7717cb97",
      "gpgkey": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBFturGcBEACv0xBo91V2n0uEC2vh69ywCiSyvUgN/AQH8EZpCVtM7NyjKgKm\nbbY4G3R0M3ir1xXmvUDvK0493/qOiFrjkplvzXFTGpPTi0ypqGgxc5d0ohRA1M75\nL+0AIlXoOgHQ358/c4uO8X0JAA1NYxCkAW1KSJgFJ3RjukrfqSHWthS1d4o8fhHy\nKJKEnirE5hHqB50dafXrBfgZdaOs3C6ppRIePFe2o4vUEapMTCHFw0woQR8Ah4/R\nn7Z9G9Ln+0Cinmy0nbIDiZJ+pgLAXCOWBfDUzcOjDGKvcpoZharA07c0q1/5ojzO\n4F0Fh4g/BUmtrASwHfcIbjHyCSr1j/3Iz883iy07gJY5Yhiuaqmp0o0f9fgHkG53\n2xCU1owmACqaIBNQMukvXRDtB2GJMuKa/asTZDP6R5re+iXs7+s9ohcRRAKGyAyc\nYKIQKcaA+6M8T7/G+TPHZX6HJWqJJiYB+EC2ERblpvq9TPlLguEWcmvjbVc31nyq\nSDoO3ncFWKFmVsbQPTbP+pKUmlLfJwtb5XqxNR5GEXSwVv4I7IqBmJz1MmRafnBZ\ng0FJUtH668GnldO20XbnSVBr820F5SISMXVwCXDXEvGwwiB8Lt8PvqzXnGIFDAu3\nDlQI5sxSqpPVWSyw08ppKT2Tpmy8adiBotLfaCFl2VTHwOae48X2dMPBvQARAQAB\ntDFGZWRvcmEgKDMwKSA8ZmVkb3JhLTMwLXByaW1hcnlAZmVkb3JhcHJvamVjdC5v\ncmc+iQI4BBMBAgAiBQJbbqxnAhsPBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAK\nCRDvPBEfz8ZZudTnD/9170LL3nyTVUCFmBjT9wZ4gYnpwtKVPa/pKnxbbS+Bmmac\ng9TrT9pZbqOHrNJLiZ3Zx1Hp+8uxr3Lo6kbYwImLhkOEDrf4aP17HfQ6VYFbQZI8\nf79OFxWJ7si9+3gfzeh9UYFEqOQfzIjLWFyfnas0OnV/P+RMQ1Zr+vPRqO7AR2va\nN9wg+Xl7157dhXPCGYnGMNSoxCbpRs0JNlzvJMuAea5nTTznRaJZtK/xKsqLn51D\nK07k9MHVFXakOH8QtMCUglbwfTfIpO5YRq5imxlWbqsYWVQy1WGJFyW6hWC0+RcJ\nOx5zGtOfi4/dN+xJ+ibnbyvy/il7Qm+vyFhCYqIPyS5m2UVJUuao3eApE38k78/o\n8aQOTnFQZ+U1Sw+6woFTxjqRQBXlQm2+7Bt3bqGATg4sXXWPbmwdL87Ic+mxn/ml\nSMfQux/5k6iAu1kQhwkO2YJn9eII6HIPkW+2m5N1JsUyJQe4cbtZE5Yh3TRA0dm7\n+zoBRfCXkOW4krchbgww/ptVmzMMP7GINJdROrJnsGl5FVeid9qHzV7aZycWSma7\nCxBYB1J8HCbty5NjtD6XMYRrMLxXugvX6Q4NPPH+2NKjzX4SIDejS6JjgrP3KA3O\npMuo7ZHMfveBngv8yP+ZD/1sS6l+dfExvdaJdOdgFCnp4p3gPbw5+Lv70HrMjA==\n=BfZ/\n-----END PGP PUBLIC KEY BLOCK-----\n"
    }
  ],
  "packages": [
    "@Core",
    "chrony",
    "kernel",
    "selinux-policy-targeted",
    "grub2-pc",
    "spice-vdagent",
    "qemu-guest-agent",
    "xen-libs",
    "langpacks-en"
  ],
  "exclude_packages": [
    "dracut-config-rescue"
  ]
}

[/usr/lib/tmpfiles.d/abrt.conf:2] Unknown user 'abrt'.
[/usr/lib/tmpfiles.d/colord.conf:1] Unknown user 'colord'.
[/usr/lib/tmpfiles.d/colord.conf:2] Unknown user 'colord'.
[/usr/lib/tmpfiles.d/cups.conf:4] Unknown user 'lp'.
[/usr/lib/tmpfiles.d/gluster.conf:2] Unknown user 'gluster'.
[/usr/lib/tmpfiles.d/httpd.conf:1] Unknown group 'apache'.
[/usr/lib/tmpfiles.d/httpd.conf:2] Unknown user 'apache'.
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
[/usr/lib/tmpfiles.d/lockdev.conf:3] Unknown group 'lock'.
[/usr/lib/tmpfiles.d/openvpn.conf:1] Unknown group 'openvpn'.
[/usr/lib/tmpfiles.d/openvpn.conf:2] Unknown group 'openvpn'.
/usr/lib/tmpfiles.d/pptp.conf:1: Line references path below legacy directory /var/run/, updating /var/run/pptp → /run/pptp; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/radvd.conf:1] Unknown user 'radvd'.
[/usr/lib/tmpfiles.d/rpcbind.conf:2] Unknown user 'rpc'.
[/usr/lib/tmpfiles.d/screen.conf:2] Unknown group 'screen'.
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system

After which it seems to hang in the dnf stage (strace: trace.log.282416.log).

Package versions: dnf-4.2.9-5.fc31.noarch

For others interested in learning more about osbuild, it would be nice to have some documentation about stages and pipelines, to get a better understanding of how it works and it’s capabilities.

Acceptance criteria:

• a high level overview of stages is included in README.md
• links to more low level documentation that exists can also be included if helpful (e.g. https://github.com/osbuild/osbuild/blob/master/stages/org.osbuild.grub2)

Follow-up from #124:

Make following changes to tree-diff:

• add metadata to added_files
• in deleted_files include only the root of deleted subtree

In order to support Azure images we need to introduce support for vpc file format in qemu assembler and we also need to align size to 1MB. For more information see this issue:
osbuild/osbuild-composer#111
and Microsoft documentation:
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-generic

This is a tracking issue for several improvements of the store:

• Support automatic cache cleanup. This will very likely mean we support a configuration on how big the store is allowed to get, and then automatically drop elements if it exceeds this. I think the first step will be to clear the entire cache whenever we hit the limit. Ultimately, we might want to consider LRU logic where we only drop the least used elements.
• Add store versioning. This will allow us to automatically clean the entire store whenever we add new metadata, etc. That is, any osbuild tool will just drop the store if its version mismatches.
• Support parallel access to the store. All operations must allow parallel readers and writers. This might require locking.
• Store metadata with every entry. Make sure this can be easily extended in the future.
• Possibly merge the source-cache with the object-cache. If we allow metadata in the store, we can just specify whether the stored content is a file or a directory.

Comments welcome! Otherwise, I will just go ahead and implement this.