I'm working through the Getting Started documentation and encountered an error when building the rpi4 image via ./run-bakery bake image pi4 build/image-pi4.img.

Running this on an M2 MacBook with the latest Docker Desktop (4.27.2 (137060)), which includes Docker Engine 25.0.3. Haven't changed any settings so both the Virtualization Framework and VirtioFS are enabled.

Build output is:

➜  rugpi-template git:(main) ./run-bakery bake image pi4 build/image-pi4.img
v0.6: Pulling from silitics/rugpi-bakery
Digest: sha256:de7e5b2aac92b093767f87cc797de91a25c9e0a26fe4b5256834d912ab68a65c
Status: Image is up to date for ghcr.io/silitics/rugpi-bakery:v0.6
 INFO baking image `pi4`
Cloning into '/project/.rugpi/repositories/fe0a146c8643a35e63802830c78f74e69ff196de'...
 INFO baking layer `customized`
 INFO baking layer `raspios-bookworm`
 INFO downloading `https://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2023-12-11/2023-12-11-raspios-bookworm-arm64-lite.img.xz`
 INFO decompressing XZ image
 INFO creating `.tar` archive with system files
 INFO extracting system files
 INFO [ 1/14] apt: update package lists {}
 INFO     - 00-install.sh
 INFO [ 2/14] apt: upgrade all packages {}
 INFO     - 00-install.sh
 INFO [ 3/14] remove unnecessary Raspberry Pi OS functionality {}
 INFO     - 00-install.sh
Removed "/etc/systemd/system/multi-user.target.wants/userconfig.service".
resize2fs_once.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable resize2fs_once
Failed to disable unit, unit unattended-upgrades.service does not exist.
Removed "/etc/systemd/system/multi-user.target.wants/sshswitch.service".
 INFO [ 4/14] disable swapping {}
 INFO     - 00-install.sh
 INFO [ 5/14] install and configure Rugpi Ctrl {"rugpi_admin": "true"}
 INFO     - 00-packages
 INFO     - 01-run.sh
 INFO     - 02-install.sh
	not a dynamic executable
Created symlink /etc/systemd/system/multi-user.target.wants/rugpi-admin.service → /lib/systemd/system/rugpi-admin.service.
 INFO [ 6/14] install state file for `/etc/fake-hwclock.data` {}
 INFO     - 00-install.sh
 INFO [ 7/14] install state file for `/root` {}
 INFO     - 00-install.sh
 INFO [ 8/14] sets the hostname {"hostname": "rugpi-template"}
 INFO     - 00-install.sh
 INFO [ 9/14] install Nginx {}
 INFO     - 00-packages
 INFO     - 01-install.sh
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable nginx
 INFO [10/14] enabling SSH {"root_authorized_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2tN+ZL8lGnx8rW+vm9ukX1uhKE7GREHitIVxIN3fVh406ksaZzY4FB7JqMqor4SBpR/Eigm6mSSE6KdwSYYC99hakLVvFUG6b6xvB7gOgre8M0JuL9XwBfaUfNln6Hl2Xirlml61JwOWa8Lh+T8mquw9OUK20tkXNPrigVKH+RMQA2V0AQrHnzo9GXMT5HEdAfaVVhL8S1inlKixiPbnvt+nWUSoKGLo+I08w5ZKI88C+saP6VqFiinp57uF0F3oqwcupZe0j6vxGuN5hFg8YGcICFnjXUAVjds8pfcf7aImvYZdp192jC9JAfzx3LzJZLn+kY9hIQkqip/tfTtp56eBb+j9i07PhrDsGiZVNOWf+YG3Cw5Ix6ltOL0dvF1/xFG7O+CGz62w8Y925ytuGgzBkVJ090eznnCjpw+lhNiNFmoqUjiJFjs/VSrqmC5bqdRrqF7YIs61uKl/EyAZaEoHZJazUFFauOjjPK0ksVbAAfqxG4nFmOG0URemSvNE= koehlma@Zaylee\n"}
 INFO     - 00-install.sh
Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable ssh
Created symlink /etc/systemd/system/sshd.service → /lib/systemd/system/ssh.service.
Created symlink /etc/systemd/system/multi-user.target.wants/ssh.service → /lib/systemd/system/ssh.service.
Removed "/etc/systemd/system/multi-user.target.wants/regenerate_ssh_host_keys.service".
Created symlink /etc/systemd/system/multi-user.target.wants/hydrate-ssh-host-keys.service → /lib/systemd/system/hydrate-ssh-host-keys.service.
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2tN+ZL8lGnx8rW+vm9ukX1uhKE7GREHitIVxIN3fVh406ksaZzY4FB7JqMqor4SBpR/Eigm6mSSE6KdwSYYC99hakLVvFUG6b6xvB7gOgre8M0JuL9XwBfaUfNln6Hl2Xirlml61JwOWa8Lh+T8mquw9OUK20tkXNPrigVKH+RMQA2V0AQrHnzo9GXMT5HEdAfaVVhL8S1inlKixiPbnvt+nWUSoKGLo+I08w5ZKI88C+saP6VqFiinp57uF0F3oqwcupZe0j6vxGuN5hFg8YGcICFnjXUAVjds8pfcf7aImvYZdp192jC9JAfzx3LzJZLn+kY9hIQkqip/tfTtp56eBb+j9i07PhrDsGiZVNOWf+YG3Cw5Ix6ltOL0dvF1/xFG7O+CGz62w8Y925ytuGgzBkVJ090eznnCjpw+lhNiNFmoqUjiJFjs/VSrqmC5bqdRrqF7YIs61uKl/EyAZaEoHZJazUFFauOjjPK0ksVbAAfqxG4nFmOG0URemSvNE= koehlma@Zaylee

 INFO [11/14] install and configure ZSH {"make_default": "true"}
 INFO     - 00-packages
 INFO     - 01-run.sh
--2024-02-15 21:47:23--  https://git.grml.org/f/grml-etc-core/etc/zsh/zshrc
Resolving git.grml.org (git.grml.org)... 202.61.209.103
Connecting to git.grml.org (git.grml.org)|202.61.209.103|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://git.grml.org/?p=grml-etc-core.git;a=blob_plain;f=etc/zsh/zshrc;hb=HEAD [following]
--2024-02-15 21:47:23--  https://git.grml.org/?p=grml-etc-core.git;a=blob_plain;f=etc/zsh/zshrc;hb=HEAD
Reusing existing connection to git.grml.org:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: '/tmp/.tmpSE5c9b/etc/zsh/zshrc'

     0K .......... .......... .......... .......... ..........  256K
    50K .......... .......... .......... .......... ..........  524K
   100K .......... .......... .......                          35.5M=0.3s

2024-02-15 21:47:24 (435 KB/s) - '/tmp/.tmpSE5c9b/etc/zsh/zshrc' saved [130117]

 INFO     - 02-install.sh
 INFO [12/14] Raspberry Pi meta recipe {}
 INFO [13/14] install static webpage {}
 INFO     - 00-install.sh
 INFO [14/14] apt: cleanup cache and delete package lists {"autoremove": "true"}
 INFO     - 00-install.sh
 INFO packing system files
 INFO creating image (size: 2465550336 bytes)
fallocate: cannot open build/image-pi4.img: No such file or directory
Error: error running command `fallocate -l 2465550336 build/image-pi4.img`: command failed with non-zero exit code 1
=== STDERR ===
fallocate: cannot open build/image-pi4.img: No such file or directory

Description

OTA update fail (e.g. switch back to the original partition) when upgrading across Rugpi versions 0.6 to 0.7.

The image is successfully downloaded, and the device tries to boot into the new partition, however the device automatically rolls back after ~20 seconds.

However, when performing an OTA update from an image that was built using the same Rugpi version, the update works (e.g. the device boots up in the new partition successfully), both for Rugpi 0.6 to 0.6, and Rupig 0.7 to 0.7 based images.

Procedure

The following procedure can be used to reproduce the problem:

1. Flash the following image to the SD card

   https://github.com/thin-edge/tedge-rugpi-image/releases/download/20240603.0849/tedge_rugpi_tryboot_20240603.0849.img.xz

2. Boot Raspberry Pi 5 device (with the new SD card)

   # rugpi-ctrl system info
   Boot Flow: tryboot
   Hot: a
   Cold: b
   Default: a
   Spare: b

3. Download the second image (built using 0.7)

   https://github.com/thin-edge/tedge-rugpi-image/releases/download/20240617.2150/tedge_rugpi_rpi-tryboot_20240617.2150.img.xz

4. Open Rugpi admin webpage, e.g. http://rpi5-abcdef.local:8088, and update the device using the second image

   https://github.com/thin-edge/tedge-rugpi-image/releases/download/20240617.2150/tedge_rugpi_rpi-tryboot_20240617.2150.img.xz

5. Wait for the update

   Observations

   • The device will try to switch partition, the fan stays on for an extended period of time, until eventually the device reboots again (unaided), and it switches back to the original partition
   • The device can't be connected to via ssh (indicating that it is probably a boot loader problem, and the OS isn't starting)

   Afterwards, the rugpi-ctrl system info output confirm that the partition rollback occurred:

   # rugpi-ctrl system info
   Boot Flow: tryboot
   Hot: a
   Cold: b
   Default: a
   Spare: b

Notes

• Repeating the procedure but using the same Rugpi 0.7 image tedge_rugpi_rpi-tryboot_20240617.2150.img.xz in both flashing the SD card and the OTA update, works.

  After the update

  # rugpi-ctrl system info
  Boot Flow: tryboot
  Hot: b
  Cold: a
  Default: a
  Spare: b

• Repeating the procedure but using the same Rugpi 0.7 image tedge_rugpi_tryboot_20240603.0849.img.xz in both flashing the SD card and the OTA update, works.

  # rugpi-ctrl system info
  Boot Flow: tryboot
  Hot: b
  Cold: a
  Default: a
  Spare: b

System information

An option to delete the overlay during the installation process to avoid mixing unexpected states together when using set-persist true.

Use-case

When using set-persist true, old state from the cold partition could cause interference when upgrading to a new image and switching back to the A partition.

graph LR
    1[A] --new image--> 2[B] --new image--> 3[A]

For example:

1. On Partition A, activate set-persist true
2. Install App1 v1 (as it was not part of the base image)
3. Install new image and switch to Partition B
4. Install new image (now containing App1 v2) and switch to partition A

The installation of App1 v1 in step 2 will override any changes to the new App1 v2 in step 4. This would prevent both compatibility challenges for the app, resulting in a non-functional app as it would mix files from the App1 v1 and v2 together.

In some situations, the state persisted when using set-persist true is only tied to the current image, and may not be relevant once a new image is applied.

In general images are used as a baseline to guarantee all of the packages work together to provide as one larger functional entity. Old stale overlay state (such as applications) should not corrupt its function.

Proposed solution

Add option when installing a new image whether any overlay partition should be deleted or not before the new image is started.

This would be a non-breaking change, and it would give the flexibility for image creators to decide whether they want to opt-in for this functionality or not.

The functionality could be exposed as a cli command in the install command:

rugpi-ctrl update install [--reset-overlay] <image>

Currently, Rugpi Bakery needs to run in a Docker container with elevated privileges (--privileged). With version 0.7, we will no longer require a loop device for building images. Unfortunately, we can still not drop this requirement as we need --bind mounts for the chroot environment in which recipes run. It would be great, if we could reduce the privileges required to run Rugpi Bakery to enable it to run in more contexts (e.g., GitLab CI).

Design Notes

Bubblewrap would be a great basis to enable rootless builds. It is also used by Mkosi. Unfortunately, Bubblewrap still does not run in arbitrary Docker containers (see containers/bubblewrap#505).

Design Proposal

Switch to using Bubblewrap and potentially allow the execution outside of Docker. For Docker, we then still need some elevated privileges but probably can set them on a more fine-grained basis.

Refactor Rugpi Bakery to support layers. Each layer may add additional customizations (recipes). Layers should be cached and shared between different variants of an image.

• Introduce recipe collections as a way to bundle multiple recipes and put the current default recipes into a raspberrypi collection. This is necessary because we only want to apply those recipes to the bottom layer.

This needs to be done before the 1.0 release as it likely introduces backwards-incompatible changes to Rugpi Bakery.

It looks like ./run_bakery looks for recipe.toml files in each subdirectory within /project/recipes within the build container. This causes the below error when the MacOS .DS_Store/ directory is also mounted into the build container.

Perhaps the bakery could ignore hidden directories in /project/recipes, or .DS_Store/ more specifically?

Status: Image is up to date for ghcr.io/silitics/rugpi-bakery:v0.6
 INFO baking image `pi4`
Error: loading library

Caused by:
    0: error reading recipe info from path `"/project/recipes/.DS_Store/recipe.toml"
    1: Not a directory (os error 20)

Is it possible to use rugpi-ctrl to indicate that the system should try booting into the cold set on next boot without calling rugpi-ctrl system reboot --spare? The use case is for scenarios in which a new image has been downloaded and installed to the cold spare, but the reboot should not occur immediately (e.g. because the user is in the middle of something). Ideally, the user could power the system off whenever they please, and the next time they power it on it'll boot into the cold spare.

In principle, it should be possible to add support for older boards without the tryboot feature by using U-Boot as an intermediate bootloader. U-Boot would then be responsible for switching between A/B partitions. This would also open up the possibility to use Rugpi with other boards than Raspberry Pi. If you want to see this happen, please give this issue a 👍.

Known Caveats

• There can only be a single config.txt and it needs to be configured to load the bootloader instead of the kernel.

Open Questions

• Consider using RAUC as a basis and implement the current tryboot switching within RAUC.
• Consider using SWUpdate.

Hi there, is there a recommended way to generate and extract a unique image ID from a built image? I'd like to build an image within a CI pipeline running in GitHub Actions. During the build, a step would create a random ID (i.e. using tr and /dev/urandom) and write it to a file within the rootfs. Then, from within the GitHub Action the ID would be extracted from the image and saved to a db somewhere.

On my machine I figure I could just mount the built image using Docker and extract the file. Or, just generate the ID file on my machine and copy it into the rootfs as part of the recipe. But I'm not sure how that would work with GitHub Actions? Figured I'd ask in case I'm missing something obvious. Thanks!

Support streaming updates. Currently, the image first needs to be completely downloaded which requires enough available space. For smaller devices, this may be prohibitively expensive.

Is it possible to specify multiple directories using [[persist]] and directory? Maybe as follows?

[[persist]]
directory=/root
directory=/somethinglese

or

[[persist]]
directory=/root

[[persist]]
directory=/somethingelse

As the root filesystem is read-only anyway, we may use Squashfs to compress it. This would leave more space for the data partition. As an additional benefit, the size of the (uncompressed) image would be reduced.

Add support for Rugpi repositories.

A repository may provide additional recipes, collections, and layers (once those are supported).

I've pulled the template from the rugpi-template repo and can bake it successfully using ./run-bakery bake image pi4 build/image-pi4.img.

I've tried adding a simple additional recipe that installs the jq package into the customized layer but get the below error when baking. Running on an M2 MacBook Air with Sonoma 14.2 and the latest version of Docker Desktop.

➜  rugpi-template git:(main) ✗ ./run-bakery bake image pi4 build/image-pi4.img
v0.6: Pulling from silitics/rugpi-bakery
Digest: sha256:026a07d0517564a027f1ac788fd17c9f61aaebac9b5dcdb254ec667c150f3d94
Status: Image is up to date for ghcr.io/silitics/rugpi-bakery:v0.6
 INFO baking image `pi4`
Error: loading library

Caused by:
    invalid architecture

I've added the recipe by appending its name to the recipes array in layers/customized.toml. The recipe has one step 00-packages that contains one line jq to install the package. Any ideas?

It should be possible to run Rugpi Bakery outside of the Docker container as this gives more flexibility.

This is a great project and I think the community will benefit a ton from it. The SBC community needs more projects like this. Have you thought about including an option for sponsoring this work (would be happy to sponsor!)

As of now, we ship updates in the form of system images that can also be directly flashed to a storage medium. While it is convenient to have only a single artifact, these images waste space and are also very inflexible when it comes to shipping updates. In particular, if we aim to eventually support delta updates, we need to develop a proper and more flexible artifact format. This issue tracks the progress of the design and implementation of an artifact format.

Design requirements

• R1: The format must support streaming updates.
• R2: The format should allow for random accesses.

Prior Art

• RAUC uses Squashfs images which are mounted by the update client via a network-backed block device using HTTP range queries: https://rauc.readthedocs.io/en/latest/reference.html#sec-ref-formats
• Mender uses tar archives.

While RAUC's approach supports random accesses, it does not allow for primitive streaming of artifacts without seeking. While Mender's approach allows streaming, it does not allow random accesses.

Design Proposal

We will be using Casync's catar file format as a basis as catar files support both streaming and random accesses. This will allow the update client to accept a stream while also giving us the flexibility to later use HTTP range queries or similar mechanisms to only retrieve certain parts of an artifact. In particular, this would also allow us to build artifacts containing multiple updates or updates in different formats, leaving it to the client which updates to install.

As with RAUC and Mender, a Rugpi update artifact contains a manifest describing the update. Ideally, this manifest would be the first file in the archive stream, followed by system images, Casync indices for the filesystem, or delta streams.

• Decide on the exact details of the artifact format.

Currently the official way to enable rugpi-ctrl state overlay set-persist true, however there does not seem to be an option to enable the setting by default during the build process.

The current workaround is to create a recipe that installs a one-shot service which enables the setting on first boot (and using the systemd first-boot-complete.target option)

Use Case

In a perfect world, set-persist true would not be required as everything the user would need would be part of the image. Or if it did not included everything, at least the directories/files that should be parked as persisted should be known at build time.

In practice, it is very difficult/impossible to known all of the directories/files that should be persisted at build time. Therefore using the set-persist true can be attractive because it allows users to customize their images to their liking (e.g. install custom applications, custom configuration files etc.)

• Install custom applications / delta updates
• Allow persistence of files without the user having to know much about the underlying persistence layer

Though it must be noted that if the user then upgrades their image and switches partition (e.g. A -> B), then any user-created state in the current partition will be lost unless if they backup/restore data from a persistent folder. However this is trivial to support (and the default in cases like mender), where the /data partition is a persisted across A/B partitions, therefore it enables user the ability to backup and restore their files if necessary (assuming they have tooling around the calling of the install new image command)

Proposed Solution

Add option in the rugpi-bakery.toml to enable set-persist true by default.

default_persist = true

I noticed that the some changes are made to the config.txt file during build. It would be nice if it were possible to add other options at some point as well (maybe some kind of ability to add lines the layer.toml?

(For now I'm just making the changes out-of-band which also works fine).

It would be nice if we could support efficient delta updates. To this end, we first need to define a proper artifact format (see issue #25). An artifact may then contain a delta encoding of a filesystem (e.g., in the VCDIFF or a similar format).

Design requirements:

• R1: There should be a mechanism to make sure that the blocks used from another filesystem indeed have the expected contents.

Design notes:

• While being an open standard with tool support (e.g., Xdelta), VCDIFF does not seem to have a built-in mechanism to check the integrity of copied blocks. We would need to somehow add this on-top, e.g., by computing the hash over the entire update and checking whether it matches our expectations.

Blockers:

• #25

Hey there - this looks like a great project that could have a big impact on those building products on RPi. Thanks for putting it out there!

Wondering whether signed update support is on the roadmap at some point? Signed updates are a must-have for those using Rugpi to ship commercial products and may also be important for hobbyists as well.

Thanks again for the work on the project and making it open source with permissive license!

I referred to the rugpi-template project and modified the layers/customized.toml to raspios-desktop (referenced from here: https://github.com/kfihihc/tedge-rugpi-core/tree/main/layers). I successfully built it using sudo ./run-bakery bake image tryboot build/image-pi5-rugpi.img. However, after flashing it to the SD card, it fails to boot. The official fan HAT of Raspberry Pi 5 keeps spinning, then stops, waits for a while, and then starts spinning again, indicating that there seems to be a problem with the boot process.

Additionally, when I referred to the rugpi-template project and built an image based on Raspberry Pi OS Lite, it could boot normally. Using rugpi-ctrl update install image-pi5-rugpi.img on a normally booted RPi OS Lite system to OTA to RPi OS Desktop results in an error message saying No enough space or such file.

Is there anything else I need to be aware of when building images based on RPi OS Desktop with Rugpi?

I got the same problem mentioned in #15.

I still find the baked image for Raspberry OS Desktop can not boot.

What I have done is:

1. install the latest version, which I believe is v0.7.4
2. use ./run-bakery init rpi-raspios to get template files
3. enter ./run-bakery shell, and directly change the url in /usr/share/rugpi/repositories/core/layers/raspios-bookworm.arm64.toml
4. run ./run-bakery image tryboot-pi4

Then the image can not boot normally.

Thanks for your help.

Different Raspberry pi devices (pi 3, 4 and 5) requires slightly different configuration options (e.g. boot_flow, include_firmware etc.)

Currently rugpi-bakery expects that the image definition is defined in the rugpi-bakery.toml file, this does not allow to create different image variants which use the same recipes, but just differ slightly...To enable this you would have to have 1 image definition per repository which becomes more difficult to manage.

Proposal

• rugpi-bakery supports providing the configuration file as an optional flag (defaulting to the existing value rugpi-bakery.toml)

Example usage

./run-bakery customize --config rugpi-bakery.pi3.yaml
./run-bakery customize --config rugpi-bakery.pi4.yaml
./run-bakery customize --config rugpi-bakery.pi5.yaml
./run-bakery customize --config rugpi-bakery.pi4.with_firmware.yaml

While it is possible to import a root filesystem built with a third-party tool into Rugpi Bakery, this requires a two-step build process for building the image. Instead, it would be great if there was a more direct integration into OpenEmbedded and Yocto. As we do not want to replicate the image building logic, it would be great if this integration could reuse Rugpi Bakery to build the image based on the filesystem built with OpenEmbedded.

Blockers:

• #27