Centos Bootc is used as the parent image for many of our other bootc images in ai-lab-recipes repo, that attempt to make pull in sizeable images, sometimes resulting in network issues like connection reset by peer. For this reason I would like to petition to bump Podman to version v5.0.0 so that we can leverage the --retry flag in podman pull.

/cc @rhatdan

$ podman run --rm -ti quay.io/centos-bootc/centos-bootc:stream9 dnf -y install rootfiles
...
  Installing       : rootfiles-8.1-31.el9.noarch                                                                                                                                                                                                                                       1/1 
error: failed to open dir root of /root/: File exists

One can work around this right now with a RUN mkdir -p /var/roothome; but we should try to fix that package to use systemd-tmpfiles instead probably.

Deploying quay.io/flightctl/flightctl-agent-centos:bootstrap with bib fails like this:

ostree container image deploy --imgref=ostree-unverified-image:dir:/tmp/tmpl16zdc_g/image --stateroot=default --target-imgref=ostree-unverified-registry:quay.io/flightctl/flightctl-agent-centos:bootstrap --karg=rw --karg=console=tty0 --karg=console=ttyS0 --karg=root=LABEL=root --sysroot=/run/osbuild/tree
error: Performing deployment: Importing: Parsing layer blob sha256:9de5a4ff2cc05f1dc30ea793a9a10bca5935ea3f60965962ac68c5b655a68b32: : Processing tar via ostree: No such file or directory (os error 2)

At first, I thought this was another echo of coreos/rpm-ostree#3827 (comment) ➡️ ostreedev/ostree-rs-ext@883a79e

Digging into the relevant layer, it's the one corresponding to this run invocation:

RUN dnf install -y flightctl-agent && \
    systemctl enable flightctl-agent.service

Digging in a bit...the first thing I notice here is that building that with podman here works fine. And yes...it looks like that same issue, because in the remotely built image we have:

hrw-r--r-- 0/0               0 usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite link to sysroot/ostree/repo/objects/cd/3e1735c45d8013153d7808e4efb834b0478016266aaf5f740c1ff0e8262119.file
hrw-r--r-- 0/0               0 usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite-shm link to sysroot/ostree/repo/objects/bf/13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187.file

But I thought we'd fixed this one....ah yes, in ostreedev/ostree-rs-ext@43e1648 and also in ostreedev/ostree-rs-ext#558

Once https://gitlab.com/redhat/centos-stream/release-engineering/pungi-centos/-/merge_requests/529 percolates (I assume we'd see this in a development compose within a day)? let's fix our base configs to require it.

This repo produces two container images. It would be preferable if it would produce only one container image while the second one is tracked in its own repo.
Eventually, the target is to migrate the fedora one to be owned by fedora so it makes sense to split the sources at some point.
As another argument, it is required to define those deliverables (container images) in different RHTAP applications due to tagging constrains.
Wouldn't it make sense to perform the split in conjunction with the applications splitting?

I have a Fitlet2 device that I'm trying to install the ELN bootc image (quay.io/centos-boot/fedora-tier-1:eln) to.

Using the F39 netinstall ISO from a USB, I used inst.ks=https://miabbott.fedorapeople.org/centos-boot.ks to point to a version of https://github.com/CentOS/centos-boot/blob/main/docs/example.ks that has my SSH public key included.

The install proceeds successfully until the "post-installation setup tasks" portion of the install.

I get the following error (transcribed from a physical monitor, so apologies for errors..):

The command 'ostree admin instutil set-kargs root=UUID=<UUID value> rw' exited with the code -4

Digging through the syslog (attached below), it looks like it could be SELinux denying the operation:

19:11:27,208 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:Running in chroot '/mnt/sysroot'... ostree admin instutil set-kargs root=UUID=1275dfc6-bad2-4498-8889-e0726e0c900d rw
19:11:27,214 NOTICE audit:AVC avc:  denied  { execute_no_trans } for  pid=4190 comm="python3" path="/usr/bin/ostree" dev="sda5" ino=34006253 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:install_exec_t:s0 tclass=file permissive=1
19:11:27,215 NOTICE kernel:audit: type=1400 audit(1700334687.213:732): avc:  denied  { execute_no_trans } for  pid=4190 comm="python3" path="/usr/bin/ostree" dev="sda5" ino=34006253 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:install_exec_t:s0 tclass=file permissive=1

I'll try again with an ELN boot ISO, in case there is SELinux policy differences that may come into play.

Tracking this here from ostreedev/ostree-rs-ext#607

This currently wants a patch to rpm-ostree...basically coreos/rpm-ostree@7857feb special cased root.transient = true, which we were going to ship but then I got talked down (correctly, probably) to just using composefs, but not transient root.

Except as determined there we need an opt-in to configure the symlinks, which we should then flip in this repository as we always want to have the container-oriented flow.

See openshift/os#1476 (comment).

$ podman run -ti --rm quay.io/centos-bootc/centos-bootc:stream9 systemctl is-enabled dnf-makecache.timer
enabled

But cc @dcantrell for his thoughts on the suggestion there.

Intalling Postgres in the latest centos-bootc-dev base container fails on the ostree commit due to content being placed in /var/lib by the rpm install. This container should include the code from ostreedev PR #569

Error from podman build:

STEP 3/3: RUn ostree container commit
Found file: var/lib/pgsql/.bash_profile
error: Found content in var
Error: building at STEP "RUN ostree container commit": while running runtime: exit status 1

Simple containerfile to reproduce

FROM quay.io/centos-bootc/centos-bootc-dev:stream9

RUN dnf install -y postgresql-server && \
    rm /var/log/*.log /var/lib/dnf -rf

RUN ostree container commit

Output of podman images
quay.io/centos-bootc/centos-bootc-dev stream9 96188766724f About an hour ago 1.45 GB

ostree info from inside container:
bash-5.1# ostree --version

libostree:
 Version: '2023.9'
 Git: f9d013632d88db3a2a89a8c4924417b3cfbdabec
 DevelBuild: yes
 Features:
  - inode64
  - libcurl
  - gpgme
  - composefs
  - ex-fsverity
  - libarchive
  - selinux
  - openssl
  - sign-ed25519
  - libmount
  - systemd
  - devel
  - p2p

bash-5.1# rpm -qa | grep ostree
ostree-libs-2023.8.54.g19cd8cf1-%autorelease.x86_64
ostree-2023.8.54.g19cd8cf1-%autorelease.x86_64
rpm-ostree-libs-2024.1.23.gefe57a65-1.el9.x86_64
rpm-ostree-2024.1.23.gefe57a65-1.el9.x86_64

Following ostreedev/ostree@f81b9fa we now have improved semantics for including content in /var in the container image by default.

There is still an inherent tension between this and systemd-tmpfiles.

However, we already have a few things in the image like /var/tmp because without it too many things fail.

Let's go ahead and extend that set to at least:

• /var/roothome
• /var/home
• /var/log/journal

ssh $SSH_ARGS "$SSH_HOST" podman run --mount type=bind,source=$BUILD_DIR/tmp,target=/var/tmp,relabel=shared --privileged -e CONTEXT="$CONTEXT" -e IMAGE_FILE="$IMAGE_FILE" -e IMAGE="$IMAGE" -e IMAGE_EXPIRES_AFTER="$IMAGE_EXPIRES_AFTER" -e COMMIT_SHA="$COMMIT_SHA" --rm -v "$BUILD_DIR/workspaces/source:/workspace/source:Z" -v $BUILD_DIR/scripts:/script:Z --user=0 "$BUILDER_IMAGE" /script/script-build.sh
time="2024-01-11T09:43:00-05:00" level=warning msg="Failed to get rootless runtime dir for DefaultAPIAddress: lstat /run/user/1003: no such file or directory"
error creating temporary file: No such file or directory

Not clear to me what the temporary file we're talking about here. I don't believe anything changed in this codebase; something changed in the rhtap environment

This is clearly not our fault but tracking it here since it affects this repo at least

Failed to create PVC for PipelineRun centos-bootc-tenant/centos-bootc-on-pull-request-cbznq Workspaces correctly: failed to create PVC for PipelineRun centos-bootc-on-pull-request-cbznq: [failed to create PVC pvc-9e7a66e281: persistentvolumeclaims "pvc-9e7a66e281" is forbidden: exceeded quota: storage, requested: count/persistentvolumeclaims=1, used: count/persistentvolumeclaims=30, limited: count/persistentvolumeclaims=30, failed to create PVC pvc-a3eed7f06b: persistentvolumeclaims "pvc-a3eed7f06b" is forbidden: exceeded quota: storage, requested: count/persistentvolumeclaims=1, used: count/persistentvolumeclaims=30, limited: count/persistentvolumeclaims=30]

Today these projects contain logic to grow the rootfs by default:

• cloud-init
• Ignition (as integrated in fcos and derivatives)
• systemd-repart

There may be others. If one generates an AMI (e.g.) from our generic base image, then the rootfs won't be grown.

Now of course...ideally one can specify the rootfs default size in the container (e.g. instead of dynamically choosing 100G at instance creation time, I can write a partitioning config in the container). See containers/bootc#287

But...a lot of users are probably going to try just deploying our stock base image as an AMI generated via bib, which hardcodes 10G right now. (see above issue for some discussion)

It looks to me like systemd-repart doesn't directly support this (its Type=root seems to want to dispatch via the discoverable partitions spec, but we can't rely on that), but I can't imagine it'd be too hard to add. I think conceptually cloud-init would be OK if we grew the rootfs before it did.

Currently we just push to an effective :latest tag, and we've had multiple instances already for which older tags would have been extremely useful.

Fixing our version field

The version field we inject in the container is written by rpm-ostree and looks like e.g. "version": "39.20240123.0". However...because we're not passing the previous image when doing a build right now, that last component is always .0. That's the first problem.

My strawman here is...we change the build process to count the number of git commits for the current day and that becomes the last digit.

Rotating tags

Let's change the release process to:

• Also push the version field as a tag, but with quay tag expiration such that it only lasts...two weeks? Maybe three?

bootc-image-builder will soon gain the ability of building an installer ISO for bootable container.

This comes with an interesting challenge: For Fedora, we use a Fedora installer, but for CentOS, we should probably use a CentOS one.

However, in order to find out what distribution is inside the bootable container, we would currently need to introspect the image to read /etc/os-release. This is surely possible, but it would be quite convenient to just read it from the labels. We fetch the labels much sooner in the image build process before fetching the whole container image. Generally speaking, the ability to know which distribution is inside the container based on its labels feels quite natural to me.

Would it be possible to add a label to the containers with the distribution ID? My suggestion is to just take ID from /etc/os-release and make it a label. VERSION_ID might be also nice.

cc @supakeen @mvo5

Recently we added subscription-manager.

This had the side effect (not noticed for a bit) of breaking one of our prime layered image builds over here - that build cleans out some cruft that dnf leaves behind.

However, the presence of subscription-manager causes even more cruft in e.g. /var/lib/rhsm; the full list now looks like:

# find /var
/var
/var/log
/var/log/dnf.log
/var/log/dnf.librepo.log
/var/log/dnf.rpm.log
/var/log/rhsm
/var/log/rhsm/rhsm.log
/var/log/hawkey.log
/var/lib
/var/lib/rhsm
/var/lib/rhsm/repo_server_val
/var/lib/rhsm/repo_server_val/redhat.repo
/var/lib/rhsm/productid.js
/var/lib/dnf
/var/lib/dnf/history.sqlite
/var/lib/dnf/history.sqlite-wal
/var/lib/dnf/history.sqlite-shm
/var/cache
/var/cache/dnf
/var/cache/dnf/expired_repos.json
/var/cache/dnf/.gpgkeyschecked.yum
/var/cache/dnf/copr-coreos-continuous-7094aa3c01d6abf5
/var/cache/dnf/copr-coreos-continuous-7094aa3c01d6abf5/repodata
/var/cache/dnf/copr-rhcontainerbot-bootc-49e4087ec07cf6f6
/var/cache/dnf/copr-rhcontainerbot-bootc-49e4087ec07cf6f6/repodata
/var/cache/dnf/eln-baseos-664ae9ea2425043f
/var/cache/dnf/eln-baseos-664ae9ea2425043f/repodata
/var/cache/dnf/eln-baseos-664ae9ea2425043f/packages
/var/cache/dnf/eln-appstream-26495fed878056ba
/var/cache/dnf/eln-appstream-26495fed878056ba/repodata
/var/cache/dnf/eln-crb-8a152f670da51e1d
/var/cache/dnf/eln-crb-8a152f670da51e1d/repodata
/var/cache/dnf/eln-extras-5ea2c05740c8bbf5
/var/cache/dnf/eln-extras-5ea2c05740c8bbf5/repodata
/var/cache/dnf/packages.db
/var/cache/dnf/tempfiles.json
/var/cache/ldconfig
/var/cache/ldconfig/aux-cache

See #132

rpm-ostree kargs --append='audit=0' is not working:

podman build -t quay.io/flightctl/flightctl-agent:latest -f packaging/Containerfile.fedora ./
STEP 1/13: FROM quay.io/centos-bootc/fedora-bootc:eln
STEP 2/13: COPY rpmbuild/RPMS/x86_64/flightctl-agent-0.0.1-1.el9.x86_64.rpm /tmp/
--> Using cache d6c77d8351a0e8777c7786cf4218fd16f120ec240b778a00bfa69eb93106db99
--> d6c77d8351a0
STEP 3/13: COPY packaging/flightctl-custom-assets/flightctl_rsa.pub /usr/etc-system/root.keys
--> Using cache e9d47ae7f9756a2a45a3a803f5817599f0a4c8f8d166fade365bfd5e06d3669f
--> e9d47ae7f975
STEP 4/13: RUN touch /etc/ssh/sshd_config.d/30-auth-system.conf;     mkdir -p /usr/etc-system/;     echo 'AuthorizedKeysFile /usr/etc-system/%u.keys' >> /etc/ssh/sshd_config.d/30-auth-system.conf;     chmod 0600 /usr/etc-system/root.keys
--> Using cache f094fb7d73fbc90d709f1d43dffc5b9b44f4739d6fae3b7f44018a48dbcb1cb7
--> f094fb7d73fb
STEP 5/13: VOLUME /var/roothome
--> Using cache 5c199e0456b55e4feb83f4bbd7afa08ef3bb8e360b141f6e203c793b1f15403b
--> 5c199e0456b5
STEP 6/13: ADD packaging/flightctl-custom-assets/config.yaml /etc/flightctl/
--> Using cache 08068ad64e01a17b5deaf1f17a5b111441b081994aa19341c1430f70c44b1935
--> 08068ad64e01
STEP 7/13: ADD packaging/flightctl-custom-assets/ca.crt /etc/flightctl
--> Using cache 6cf50e3e940d43a151d8df1efa69adb4f1ab78be5f278bba851b7bc4791ae076
--> 6cf50e3e940d
STEP 8/13: ADD packaging/flightctl-custom-assets/client-enrollment.* /etc/flightctl/
--> Using cache 467f88db436db8c62bf93775f271861275679c2e5ec8bdbed7f4e10eaf9cbb44
--> 467f88db436d
STEP 9/13: RUN rpm-ostree install -y /tmp/flightctl-agent-0.0.1-1.el9.x86_64.rpm
--> Using cache 51a5903e3b71a19a548282e35e933c153d03470071b02525b622877e3e5b2c69
--> 51a5903e3b71
STEP 10/13: RUN ln -s /usr/lib/systemd/system/podman.socket /usr/lib/systemd/system/multi-user.target.wants/
--> Using cache 886a0421c6f532094be9c8298f451e63e241c59316ad61907e06e6c6187650b6
--> 886a0421c6f5
STEP 11/13: RUN ln -s /usr/lib/systemd/system/flightctl-agent.service /usr/lib/systemd/system/multi-user.target.wants/
--> Using cache e83c1f9965b89846251ea2b1e4df3c969a52c9984e7f42c50e855035ee39fa79
--> e83c1f9965b8
STEP 12/13: RUN rpm-ostree kargs --append=audit=0
error: This system was not booted via libostree.
Currently, most rpm-ostree commands only work on ostree-based host systems.

Error: building at STEP "RUN rpm-ostree kargs --append=audit=0": while running runtime: exit status 1

This is the containerfile:

FROM quay.io/centos-bootc/fedora-bootc:eln

COPY rpmbuild/RPMS/x86_64/flightctl-agent-0.0.1-1.el9.x86_64.rpm /tmp/

COPY packaging/flightctl-custom-assets/flightctl_rsa.pub /usr/etc-system/root.keys
RUN touch /etc/ssh/sshd_config.d/30-auth-system.conf; \
    mkdir -p /usr/etc-system/; \
    echo 'AuthorizedKeysFile /usr/etc-system/%u.keys' >> /etc/ssh/sshd_config.d/30-auth-system.conf; \
    chmod 0600 /usr/etc-system/root.keys
VOLUME /var/roothome

ADD packaging/flightctl-custom-assets/config.yaml /etc/flightctl/
ADD packaging/flightctl-custom-assets/ca.crt /etc/flightctl
ADD packaging/flightctl-custom-assets/client-enrollment.* /etc/flightctl/

RUN rpm-ostree install -y /tmp/flightctl-agent-0.0.1-1.el9.x86_64.rpm
RUN ln -s /usr/lib/systemd/system/podman.socket /usr/lib/systemd/system/multi-user.target.wants/
RUN ln -s /usr/lib/systemd/system/flightctl-agent.service /usr/lib/systemd/system/multi-user.target.wants/
RUN rpm-ostree kargs --append='audit=0'
RUN ostree container commit

Any clue why? The image indeed is OSTree based.

Going via a default bootc install or bib, we don't run the preset process.

However, anaconda does.

We're going to need to ensure that all of our units (such as the autoupdate one) are added to the relevant presets.

See ostreedev/ostree#3193

Unclear which component owns the best fix, but I think short term we should ship a systemd unit that edits the anaconda-generated fstab on first boot. (We can try to patch anaconda but…messy)

I'm using a Fedora 39 boot ISO and example.ks from the instructions, and trying to install a container made from the rhel nightlies. Anaconda finishes the install just fine, but the UEFI firmware doesn't recognize /dev/sda as bootable afterwards.

I'm not sure if this is related to grub2/bootupd or if some of the additional files in /boot are missing.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• chore(deps): update auto merged updates (actions/checkout, fedora-bootc, maxbrunet/pre-commit-renovate)

Detected dependencies

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update pre-commit hook maxbrunet/pre-commit-renovate to v37.74.2

Detected dependencies

A toplevel goal of this project is to not fork the underlying OS. Contrast with e.g. https://github.com/coreos/fedora-coreos-config/ which carries its own lockfiles, which creates divergence.

Today, the fedora-derivatives tooling uses "composes" which are basically versioned directories written to NFS-like storage and served by HTTP with custom JSON metadata.

It'd be useful for us to have our version number (which is reflected both in /usr/lib/os-release and the standard org.opencontainers.image.version label) include the underlying compose version.

rpm-ostree supports a mutate-os-release key which is intended for this. I think what may work best is if we have a new yaml file like:

$ cat versioning.yaml
variables:
  composeversion: "20240113.d.0"
automatic-version-prefix: "${releasever}.${composeversion}"

or so? Then we'd need to overwrite composeversion.yaml in git too. Hmm, maybe we don't need ${releasver} here at all. Yeah...let's just simplify this to:

variables:
  composeversion: "20240113.d.0"
automatic-version-prefix: "${composeversion}"

Hey guys,

I'm trying to create a new SSH key, and I used the following steps to try and create it.

ssh-keygen -t ed25519 -C "[email protected]"
I then entered the file which I wanted to save it, which is under coding, but when it asked me to enter in a passphrase, it won't let me type anything. Do you know what's going on?

In my Containerfile I had to execute

rpm --setcaps shadow-utils

To fix the problem. We have seen similar problems with file caps missing in fedora images.

Or failing that, quay.io/centos-boot/centos-boot-tier1:stream9

We also need to put ELN someplace. Maaaybe short term it's easier to shortcut this and create quay.io/centos-boot/fedora-tier1:eln too there...

We created some documentation here on how to create a Containerfile: https://github.com/containers/podman-desktop-extension-bootc/blob/main/docs/containerfile_guide.md

Which should instead be in this repo.

Some of the points in the document is outdated, but the examples and the explanations are beneficial in order to get others understanding the concepts of building a "bootc os container"

We've been having a chat about switching to running via systemd by default, i.e. we do

CMD /sbin/init
STOPSIGNAL SIGRTMIN+3

in the base image.

I was a little uncertain about it at first...hard to describe why, it may be just a lack of familiarity with working with containers in that flow. Perhaps it just makes sense.

This is a tracking issue for the discussion on adding greenboot to this base image, in order to trigger auto rollback once the OS update failed (on some checkup).

I have the following Containerfile:

FROM quay.io/flightctl/flightctl-agent-centos:bootstrap

ADD etc etc

RUN rm -rf /opt && \
    mkdir -p /opt/crio
RUN dnf install -y microshift && \
    systemctl enable microshift.service

RUN dnf install -y kernel-5.14.0-427.14.1.blue1.el9_4

ADD systemd-shutdown /usr/lib/systemd/
RUN rm -rf /opt && ln -s /var /opt
RUN [ ! -L /var/run ] && rm -rf /var/run && ln -s /run /var/

/etc contains a repo where the kernel RPM ( kernel-5.14.0-427.14.1.blue1.el9_4 ) is stored. The container image seems to be built successfully. This image is created to be the upgrade of a base one with a very simple Container file:

FROM quay.io/centos-bootc/centos-bootc:stream9
ADD etc etc

RUN dnf install -y flightctl-agent && \
    systemctl enable flightctl-agent.service

## Add your flightctl configuration and certificates
ADD config.yaml /etc/flightctl/
ADD ca.crt /etc/flightctl/certs/
ADD client-enrollment.* /etc/flightctl/certs/

Using bootc upgrade, pulls the new image (the first Containerfile) and after reboot, I see the two ostree options in grub. However, the staged one that is booting now fails. If you enter in the kernel arguments, the initrd line is missing, and the boot partition does not contain the initramfs disk as it should.

total 8
drwxr-xr-x. 3 root root 4096 Apr 30 15:43 default-28af0175516946659bb525f36bc0fff88ef105a2b4471e1f7ecf7638ea5b57bf
drwxr-xr-x. 3 root root 4096 Apr 30 13:17 default-d341b610f2c5af9c192bccd8453fe5dbe1c2f797a5cfdb683f3222fcef4e6144
[root@localhost ~]# ll /boot/ostree/default-28af0175516946659bb525f36bc0fff88ef105a2b4471e1f7ecf7638ea5b57bf/
total 11664
drwxr-xr-x. 11 root root     4096 Apr 30 15:43 dtb
-rwxr-xr-x.  1 root root 11939803 Apr 30 15:43 vmlinuz-5.14.0-427.14.1.blue1.el9_4.aarch64
[root@localhost ~]# ll /boot/ostree/default-d341b610f2c5af9c192bccd8453fe5dbe1c2f797a5cfdb683f3222fcef4e6144/
total 47416
drwxr-xr-x. 11 root root     4096 Apr 30 13:17 dtb
-rw-r--r--.  1 root root 36562723 Apr 30 13:17 initramfs-5.14.0-437.el9.aarch64.img
-rwxr-xr-x.  1 root root 11981373 Apr 30 13:17 vmlinuz-5.14.0-437.el9.aarch64```


The new custom Kernel RPM does have contain the ramdisk, and it can be installed in a standard RHEL with no issue. 

Is there anything else we should add to the Containerfile? Is this a feature gap in bootc?

Thanks for your support!

See:

• ostreedev/ostree#1469
• https://pagure.io/workstation-ostree-config/pull-request/505

See containers/bootc#368 (comment) ➡️
coreos/rpm-ostree#4845

podman run quay.io/centos-bootc/centos-bootc:stream9 rpm -qV shadow-utils | grep /usr/bin/new.*map
.......TP /usr/bin/newgidmap
.......TP /usr/bin/newuidmap

If you do a
RUN rpm --setcaps shadow-utils
It fixes the issue. I think the tooling that builds the image is dropping file capabilties.

We merged a PR, but the image hasn't shipped. I can't access the activity tab in the build UI either, it just spins.

Wait...it seems like the image got reverted. The time stamp is now from Mar 15, but going back to an earlier version (e.g. stream9-1710868373) I see a newer date. And we definitely had a newer version for some period of time yesterday.

Something went wrong with the release pipeline?

Upon first reboot after installing the latest fedora-bootc image with bootc install to-disk --block-setup tpm2-luks /dev/diskX the LUKS encrypted root device fails to unlock via TPM. Eventually dracut times out and drops into a rescue shell in the initrd. The cryptsetup unit fails with a Current policy digest does not match stored policy digest, cancelling TPM2 authentication attempt. error. Further, an error of No passphrase or recovery key registered is also printed.

Looks like between tags eln-1710868505 and eln-1711401621 the fedora-bootc image began exhibiting this failure with the systemd-cryptsetup units on boot after install. It was working in the earlier build but now does not. I noticed a similar issue with a custom Fedora 39 based image (see containers/bootc#421) build which also failed to unlock the LUKS root via TPM on reboot after install.

Between the two builds some packages were updated. None of the ones I've looked at seem like obvious culprits though. It seems like this issue was most likely something changing in a package update?

I stumbled upon this project which I find really interesting!

As far as I understood it, this is just a regular CentOS-Stream container image including a kernel.

I wanted to experiment with installing it. So I installed Fedora Sway Atomic in a VM, after install exited to TTY immediately, and rebased to it.

rpm-ostree rebase ostree-unverified-registry:quay.io/centos-bootc/centos-bootc-dev:stream9

Everything seemed to work well, and the boot target shows correctly, but booting it in grub just results in a black screen, nothing else.

Is the image ready for usage like that? What would a correct deployment look like (apart from installing it with a dedicated anaconda configuration)?

Is this the base of future CentOS-Stream Atomic using rpm-ostree? If not, I suppose it could be hacked to mirror current Fedora Atomic systems, add rpm-ostree and work like one.

Thanks!

Right now Anaconda doesn't do this, even with the new bootupd flow.

This isn't immediately fatal or anything, but not involving os-prober etc. is a big reason why we have static configs at all.

If Anaconda used bootc install-to-filesystem this issue would go away of course.

We have similar issues with bootc-image-builder which also doesn't use bootupd yet (xref osbuild/bootc-image-builder#18 )

Ultimately this is a messy topic...either we need ostree to detect bootupd has static configs, or bootupd needs to tell ostree it doesn't need to run grub.

Let's move osbuild/bootc-image-builder#146 (comment) here - what I'm seeing is that running the eln image through bib doesn't boot, lots of selinux errors.

Checking. Probably a GC issue.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update auto merged updates (fedora-bootc, maxbrunet/pre-commit-renovate)
• chore(deps): update pre-commit hook pre-commit/pre-commit-hooks to v4.6.0

Detected dependencies

We're having a discussion about potentially moving back to somewhere underneath https://gitlab.com/CentOS/ - can folks who have an opinion on this provide them here?

At a minimum a 👍 or 👎 reaction to this issue.

See containers/bootc#179 which wants containers/bootc#256

This relates to containers/bootc#128 - but isn't quite the same thing. Let's use this as a tracker for supporting "nesting" container images.

We should ideally support something like this:

FROM quay.io/centos-bootc/centos-bootc:stream9
RUN podman --storage-driver=vfs --root=/usr/share/containers/storage pull <someimage>
COPY somecontainer.container /usr/share/containers/systemd

Where somecontainer.container is a podman systemd unit that also uses:

PodmanArgs=--root=/usr/share/containers/storage

The reason I mentioned --storage-driver=vfs is to avoid overlayfs and nested whiteouts...I think as of recent overlayfs this is supported at runtime, but...I can't make a whiteout in a default podman run invocation; I think the device cgroup may be coming into play?

$ cat Containerfile
FROM quay.io/centos/centos:stream9
RUN mknod somewh c 0 0
$ podman build -t localhost/test .
STEP 1/2: FROM quay.io/centos/centos:stream9
STEP 2/2: RUN mknod somewh c 0 0
mknod: somewh: Operation not permitted
Error: building at STEP "RUN mknod somewh c 0 0": while running runtime: exit status 1
$

Even if we could make the whiteout, I think we'd run into problems because there's no standard for nesting them at the OCI level. Also xref https://www.spinics.net/lists/linux-unionfs/msg11253.html