coreos / coreos-assembler Goto Github PK

As best I can tell...it's like OpenShift builds aren't actually copying the .git directory. We're doing COPY . . - this works with both Docker and podman.

Hmm. I wonder if we're hitting openshift/imagebuilder@084da0f

originally reported over in: openshift/os#311 by luca, copied here:

I'm not sure where in the pipeline we are baking in and recording kargs, but it looks rpm-ostree kargs manipulation drops some of them. In the example below, an --append=foo=bar results in losing coreos.oem.id=ec2.

$ ssh core@vm "sudo cat /proc/cmdline"
BOOT_IMAGE=/ostree/rhcos-ea868b22bf773dc384da77e76f66e9c51ae821f86df47d1fdd9ce62d0bebb04c/vmlinuz-3.10.0-953.el7.x86_64
no_timer_check console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 rd.neednet=1 rw coreos.firstboot=detected ip=eth0:dhcp
root=UUID=2efac7b2-3518-43ca-9e90-40649c28f7a0
ostree=/ostree/boot.0/rhcos/ea868b22bf773dc384da77e76f66e9c51ae821f86df47d1fdd9ce62d0bebb04c/0
coreos.oem.id=ec2

$ ssh core@vm "sudo rpm-ostree kargs --reboot --append=foo=bar"

$ ssh core@vm "sudo cat /proc/cmdline"
BOOT_IMAGE=/ostree/rhcos-ea868b22bf773dc384da77e76f66e9c51ae821f86df47d1fdd9ce62d0bebb04c/vmlinuz-3.10.0-953.el7.x86_64
no_timer_check console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 rd.neednet=1 rw ip=eth0:dhcp
root=UUID=2efac7b2-3518-43ca-9e90-40649c28f7a0
ostree=/ostree/boot.1/rhcos/ea868b22bf773dc384da77e76f66e9c51ae821f86df47d1fdd9ce62d0bebb04c/0
foo=bar

gf-oemid should also modify the BLS config file

When I am running a coreos-assembler container from quay.io, the only identifier I have is the image ID.

I should be able to do podman inspect and get this information.

Maybe even a coreos-assembler version command?

This came up in the review of #270 and rather than litter that review with extraneous comments, let's move it here.

Shortlist of things to implement:

• document coding standards for Python (target 3.4, using .format(), using os.path.join, etc)
• run pycodestyle (nee pep8) across the code base
• wire up TravisCI to run pycodestyle on each PR

Other input welcome. 😄

In this comment #214 (comment), @wking appears to be asking for timestamps associated with the builds in builds.json

Before I go any deeper into this hole (I've already got some rough changes made), does this makes sense to implement now? And, more importantly, did I understand the ask correctly?

What I've got so far produces a JSON that now looks like this:

{                                                                
  "builds": [                                                               
    {                                                                                                  
      "29.12": {                                                            
        "timestamp": "2018-11-14 16:48:56+00:00"
      }                                                                     
    },                                 
    {                                                                  
      "29.11": {                                            
        "timestamp": "2018-11-13 16:10:58+00:00"                   
      }                                                        
    },                                                       
    {                                                                                        
      "29.9": {                                                                              
        "timestamp": "2018-11-06 04:38:21+00:00"
      }
    }
  ]                                                                                    
}                            

WDYT?

Today c-a defaults to handling "local development". I am working on orchestrating it via Jenkins and storing builds in S3. This issue is about the design of the latter.

Currently I have a script which pulls the builds.json and then fetches the meta.json from the previous build - basically setting up enough of a "skeleton" that c-a can start a new build.

One approach we could take is to offer this as a subcommand; something like storage-s3-pull or so?
Basically, we don't try to directly integrate this into a high level flow, just make it something that can be invoked by an orchestrating pipeline.

On a related topic, the Red Hat CoreOS builds use the oscontainer subcommand in this way right now.

Despite what the README says, this container image does not ship imagefactory binary in PATH. I've read somewhere else that the image-conversion step is usually performed outside of the container.
As a casual observer, a couple of questions:

• is the image-conversion step supposed to be performed inside a (this?) container?
• can I just go on and add/use imagefactory to this container, or was it left out on purpose (and why)?

Add an option to the build command to output an initramfs.img and coreos.img to serve over PXE. This would support the https://github.com/coreos/matchbox project which would lead to tectonic integration. I'd be more than willing to contribute to this.

EDIT: I see a lot of parallels with https://github.com/weldr/lorax/blob/master/docs/livemedia-creator.rst

I keep hitting an issue where I build a FCOS image just fine using coreos-assembler but the resulting image gets MCS labels on the file so when I try to run a second container (i.e. coreos-assembler run) I get Could not open 'builds/29/fedora-coreos-29-qemu.qcow2': Permission denied.

here are the labels on the file:

[dustymabe@media srv-coreos]$ ls -lZ builds/29/fedora-coreos-29-qemu.qcow2
-rw-r--r--. 1 1000 1000 system_u:object_r:container_file_t:s0:c255,c505 1905655808 Jan 22 17:07 builds/29/fedora-coreos-29-qemu.qcow2

So I have system_u:object_r:container_file_t:s0:c255,c505 context. I then execute a podman command with a volume mount with :z for relabeling with sharing contexts and when that is done I get system_u:object_r:container_file_t:s0 context.

[dustymabe@media srv-coreos]$ sudo podman run -it --rm -v $PWD:/foo:z --entrypoint /bin/bash registry.fedoraproject.org/fedora:28
[root@d48949e7d73c /]#
[root@d48949e7d73c /]# exit
exit
[dustymabe@media srv-coreos]$ ls -lZ builds/29/fedora-coreos-29-qemu.qcow2 
-rw-r--r--. 1 1000 1000 system_u:object_r:container_file_t:s0 1905655808 Jan 22 17:07 builds/29/fedora-coreos-29-qemu.qcow2

so it removed the MCS labels and now I can coreos-assembler run.

How can I set the contexts on the directory so that the MCS labels never get applied?

Not too sure if I should of opened this up here or https://github.com/coreos/fedora-coreos-config

So my scenario is that I've built the Fedora CoreOS image. I've got the .qcow2 image, and now I'm hosting it on a libvirt / KVM box up in the cloud.

The problem is getting SSH access. Obviously OpenSSH is installed and accessible.

Obviously console login won't work: https://github.com/coreos/fedora-coreos-config/blob/master/image.ks#L22

and a user 'core' has been created: https://github.com/coreos/fedora-coreos-config/blob/master/fedora-coreos-base.yaml#L111

Currently coreos-assembler run will self-inject an SSH key into the image and it's accessible that way.

My question is:

Where do I inject my public SSH key? With the original CoreOS you could provide this in a cloud config / initial config. Should I add this to post-process? https://github.com/coreos/fedora-coreos-config/blob/master/fedora-coreos-base.yaml#L36 or am I missing something (such as an ignition config or kickstart config?)

As per discussion, we should look into moving the mantle bits to their own container. They aren't used to build the image; only for uploading, releasing, testing, etc.

Splitting them out would have the following advantages:

• Logically separating image/ostree creation and uploading/testing/releasing
• Reducing the number of container rebuilds since new commits to mantle will not affect this container
• Reduction in container size

There are of course a few disadvantages:

• Now we need two containers to do a release
• Total container size will likely increase (since there are two needed)

It was suggested that we move the building of the container image to the quay.io registry, but I don't know the repercussions of building the image on a new registry.

What are the benefits and drawbacks to building on Quay? Please discuss!

cc: @lucab @cgwalters @ashcrow

Currently, c-a outputs a "traditional" OSTree repository. There is some discussion as to what we do for FCOS.

For Red Hat CoreOS, we are going to be delivering ostree-in-container, or "oscontainer" for short. I would like the assembler to gain a configuration file (more YAML) that allows the builder to specify output formats (we also probably want this for image output types).

Something like:

filename=buildcfg.yml

output: oscontainer|ostree-repo|rojig
images:
  - qemu
  - openstack
  - iso

?

The messy part about oscontainer is dealing with container images inside a container.

We currently wrap the architecture name into a board for historical reasons, and sometimes we treat it as a per-platform option. Cleanups are needed.

I've been thinking about #184
Right now the data model is pretty simple. If we add tagging it gets more complex.

One thing I've been thinking about, and this is inspired by previous discussions around rpm-md - what if we took a page from Rust's cargo and stored our builds.json as well as the meta.json in git?

Then our tags would literally just be git tag.

Another approach is to store builds in ostree: #189 (comment)

(However today ostree doesn't have tags as such)

The main reason I want to do this is to retain some history of builds even after we do some pruning in production.

I'm trying to just get through a successful build. My apologies if this is another oops.

The host I'm running on is a Fedora 29 (upgraded from 28 today) VM on ESXi, with virtualization passthrough turned on (/dev/kvm is available). After doing a clean init, I run coreos-assembler build per the docs.

The build produces a qcow2 image from virt-install and proceeds to run gf-oemid, where it fails.

I've attached the full build log.

+ guestfish --remote -- run
libguestfs: error: appliance closed the connection unexpectedly.
This usually means the libguestfs appliance crashed.

coreos-build.log

A local developer workflow probably just involves running coreos-assembler build over and over again. The problem with this is that build passes --cache-only to rpm-ostree compose tree and thus the rpm repo-metadata won't get refreshed and no new rpms will get pulled.

We should probably think about the design a bit and have stages stack if it makes sense. I keep going back to the rpm-build -ba thinking. ba stands for build all or run all stages. bb is for build binary package (after doing the %prep, %build, and %install stages). bp executes the %prep stage. bc executes the %build stage after doing %prep. etc..

https://linux.die.net/man/8/rpmbuild

For coreos-assembler run it would be nice if we can print a message to the console to tell the user how to break out of the machine. I think once https://github.com/rfairley/console-login-helper-messages lands (or maybe just the pam upstream stuff is needed) we should be able to do this easily.

See also: #49 (comment) and
https://superuser.com/questions/1087859/how-to-quit-the-qemu-monitor-when-not-using-a-gui

For RHCOS prototyping out with this so far...caching the installer ISO isn't hard, but is going to complicate the build flow.

Given the likely future direction I'd like to cache the installer ISO in the container image as well, so that it's trivial to mirror everything. Will do a PR if no one objects.

This follows up to coreos/fedora-coreos-config#32 (comment) (list item 3).

console-login-helper-messages generates a motd at /run/motd.d/console-login-helper-messages.motd (see second box of output at coreos/fedora-coreos-config#32 (comment)), from this script. This motd contains information sourced from /etc/os-release, which CL has done.

It'd be good to have the generated motd displayed when logging in during coreos-assembler run, so that information can be sourced from /etc/os-release at runtime (e.g. ANSI_COLOR for the OS release, the OS version, anything else from other system files...).

Showing the generated motd would mean the static line at https://github.com/coreos/fedora-coreos-config/blob/master/fedora-coreos-base.yaml#L129 can be removed (but still keep the rest of the static information).

In order to show the generated motd, login can be configured to show a list of motds with the path to the generated motd given in addition to /etc/motd (see coreos/fedora-coreos-config#32 (comment) (list item 3)).

It'd be a later decision to carry the changes in the /etc/login.defs config forward for an FCOS release. For now, something like using Ignition to write the /etc/login.defs with the desired config can be done in the cmd-run script only.

filing this to give an update of my findings in getting coreos built on aarch64. The main issues I have hit so far are the calls to qemu-kvm, it is a x86 only binary, along with the grub config being in a different location due to EFI.

I am currently stuck on figuring out how to mount the ESP partition as it does not end up with a disk label.

Basically rpm-ostree doesn't write the --write-composejson-to if nothing changed. If it did, this would be fixed.

Would be cool to make init be completely idempotent and only actually error out if it's not purely creating something, but also deleting (in which case we'd require --force). That'd make pipelines easier as well.

Hi, IMHO one of the most exciting features of podman is the rootless execution of containers.

I have tried to share /dev/kvm using --mount=type=bind,source=/dev/kvm,destination=/dev/kvm.
This works for init and fetch, but build fails:

user@host ~/coreos % podman --log-level info run -ti --rm --net=host --userns=host --privileged -v $(pwd):/srv --workdir /srv --mount=type=bind,source=/dev/kvm,destination=/dev/kvm quay.io/coreos-assembler/coreos-assembler build
INFO[0000] running as rootless                          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
INFO[0000] running as rootless                          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
WARN[0037] Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied 
Using manifest: /srv/src/config/manifest.yaml
libostree:
 Version: '2018.10'
 Git: c5aaabe9d754d2ada17d5cc006e13105ab5cecc2
 DevelBuild: yes
 Features:
  - libcurl
  - no-http2
  - libsoup
  - gpgme
  - libarchive
  - selinux
  - openssl
  - libmount
  - devel
  - p2p
rpm-ostree:
 Version: '2018.9'
 Git: 0e24944c4ec0a45eb88169a60c96c8b76e86cd61
 Features:
  - compose
  - rust
Previous commit: none
Kickstart checksum: 93de3934c4488ab094a1b8ffce1b00ac1cf8d1c4d7091519202ceb484f2d25fc
Running: rpm-ostree compose tree --repo=/srv/repo --cachedir=/srv/cache --touch-if-changed /srv/tmp/treecompose.changed --unified-core /srv/src/config/manifest.yaml --cache-only --add-metadata-from-json /srv/tmp/build/tmp/commit-metadata-input.json --write-composejson-to /srv/tmp/build/tmp/compose.json
RPM-OSTree Version: 2018.9
No previous commit for fedora/29/x86_64/coreos
Enabled rpm-md repositories: dustymabe-ignition fedora fedora-updates fedora-updates-testing
rpm-md repo 'dustymabe-ignition' (cached); generated: 2018-11-21T22:40:42Z
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'fedora-updates' (cached); generated: 2018-11-23T02:15:41Z
rpm-md repo 'fedora-updates-testing' (cached); generated: 2018-11-23T02:31:21Z
Importing rpm-md... done
Resolving dependencies... done
Installing 388 packages:
  GeoIP-1.6.12-4.fc29.x86_64 (fedora)
...
  zlib-1.2.11-14.fc29.x86_64 (fedora)
Input state hash: 22c7dab4135b85f9d8bea7c3d4e75989a1460b566e1c60f2f5a6972d6f700cdc
Checking out packages... done
⠁ Running pre scripts... libini_config
fuse: device not found, try 'modprobe fuse' first
⠉ Running pre scripts... dbus-common
dbus-common.prein: bwrap: execvp /bin/sh: No such file or directory
fusermount: failed to unmount /tmp/rpmostree-rofiles-fuse.XODVvm: Invalid argument
Running pre scripts... done
error: Running %prein for dbus-common: Executing bwrap(/bin/sh): Child process killed by signal 1
podman --log-level info run -ti --rm --net=host --userns=host --privileged -v  11,15s user 21,99s system 54% cpu 1:00,25 total

Do you see any chance to enable rootless builds?

podman version 0.11.1.1
Arch Linux

I see this periodically, looks like a race condition:
ERROR Failed to connect socket to '/home/builder/.cache/libvirt/libvirt-sock': No such file or directory

I wonder if it'd help to pre-create /home/builder/.cache? Also relatedly, we should pre-cache the guestfish VM image with a dry run.

The fine folks over at http://github.com/coreos/ignition have a JSON schema for the Ignition configs that doubles as the spec for the project.

https://github.com/coreos/ignition/blob/master/schema/ignition.json

While I don't think have a need for a spec for this project, having schemas defined for the JSON files that are generated will help us check for regressions as we make changes to the project.

During the coreos-assembler build process on a Fedora 29 VM with nested kvm enabled, I'm getting a ton of cp: setting attributes for XXX: Operation not supported errors before the operation errors out. This is with podman, but with docker the operation goes further (It's stopping much later on, with a ModuleNotFoundError: No module named 'pytz' error -- I'll follow up on that separately).

dracut: FAILED: /usr/lib/dracut/dracut-install -D /tmp/dracut/dracut.HD3Ged/initramfs /usr/lib/dracut/modules.d/30ignition/ignition-files.service /usr/lib/systemd/system/ignition-files.service
cp: setting attributes for '/tmp/dracut/dracut.HD3Ged/initramfs/usr/lib/systemd/system/ignition-ask-var-mount.service': Operation not supported
dracut-install: ERROR: installing '/usr/lib/dracut/modules.d/30ignition/ignition-ask-var-mount.service' to '/usr/lib/systemd/system/ignition-ask-var-mount.service'
dracut: FAILED: /usr/lib/dracut/dracut-install -D /tmp/dracut/dracut.HD3Ged/initramfs /usr/lib/dracut/modules.d/30ignition/ignition-ask-var-mount.service /usr/lib/systemd/system/ignition-ask-var-mount.service
cp: setting attributes for '/tmp/dracut/dracut.HD3Ged/initramfs/lib/modules/4.19.8-300.fc29.x86_64//kernel/drivers/firmware/qemu_fw_cfg.ko.xz': Operation not supported
dracut-install: dracut_install 'qemu_fw_cfg' ERROR
dracut-install: ERROR: installing 'qemu_fw_cfg'
dracut: FAILED:  /usr/lib/dracut/dracut-install -D /tmp/dracut/dracut.HD3Ged/initramfs --kerneldir /lib/modules/4.19.8-300.fc29.x86_64/ -m qemu_fw_cfg
dracut: installkernel failed in module ignition
error: Finalizing rootfs: During kernel processing: Executing bwrap(rpmostree-dracut-wrapper): Child process killed by signal 1

There are two main parts of coreos-assembler: the os container and the build scripts. For CL these are actually seperate and the src directory gets bind-mounted into the chroot. Separating them (e.g. into seperate repos) here would have a few advantages:

• changes to the build scripts are lighter weight (don't need to build new containers for every one). This would make development of the build system easier.
• build scripts should be versioned with the thing they build. The host OS is less critical. This would better group the things that should be released together
• Make it easier to tell what are host requirements and what are steps in the build process

I was following the directions in the README until it broke with an apparent SELinux issue. I have SELinux enforcing, but I've ran it in permissive mode too with the same error. I haven't tried disabling it altogether, but that seems counter to the Fedora philosophy (and mine).

[devuser@localhost coreos]$ cd /srv/coreos
[devuser@localhost coreos]$ alias coreos-assembler='podman run --rm --net=host -ti --privileged --userns=host -v $(pwd):/srv --workdir /srv quay.io/cgwalters/coreos-assembler'
[devuser@localhost coreos]$ coreos-assembler init https://github.com/coreos/fedora-coreos-config
Trying to pull quay.io/cgwalters/coreos-assembler...Getting image source signatures
Copying blob sha256:e69e955c514f72cef9e9e7db408266a32ba48133f0e2a27d5e4b52ed1995d864
 85.99 MB / 85.99 MB [======================================================] 5s
Copying blob sha256:21600e57f2c7e6952d55f945688a9f886d5b2caebece0f4897081760f61ca73f
 160 B / 160 B [============================================================] 0s
Copying blob sha256:e398bc963f4ded7eaab2a9bc9e726560df344158d253dc148109af0628ced04a
 2.71 KB / 2.71 KB [========================================================] 0s
Copying blob sha256:f64fded7e3a7ecf43bba485a1dfc51f7b9cbdf4f9bf644d01c79fc84d41d4eb3
 1.15 KB / 1.15 KB [========================================================] 0s
Copying blob sha256:c11ad1c9e64a591c75dce813c670fc0739bcc4f604c96704e2a9eb30cea29f3a
 588.94 MB / 588.94 MB [===================================================] 30s
Copying blob sha256:19a8e73a980698ae96f543930e6e4e76e6bbfa6d0b63616c9f1ce0a5c993dc12
 3.51 MB / 3.51 MB [========================================================] 0s
Copying blob sha256:999624bd993360d23122d4753564f8ad21e98fcfb624046eb6d709bb7325340f
 169.86 MB / 169.86 MB [===================================================] 11s
Copying blob sha256:efbb6d113281db538c8a453bd9717ccf762f4252de549a8adec7b290fab8be14
 3.32 KB / 3.32 KB [========================================================] 0s
Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
 32 B / 32 B [==============================================================] 0s
Copying blob sha256:b01f865b2ec506c5b8d510c88bfc7251c54cc16c5fc4502b3b5e79f00d2eeba3
 137 B / 137 B [============================================================] 0s
Copying blob sha256:707462d70f0ee977c4e99d1dc2339e1eb7b9508690ea73cc7b442c4813ca90f6
 678 B / 678 B [============================================================] 0s
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Writing manifest to image destination
Storing signatures
umount: /sys/fs/selinux: not mounted.
[devuser@localhost coreos]$ getenforce
Enforcing

I've also tried docker, which was able to initialize but I ran into a libguestfs issue there, I'll open separately.

I've accumulated quite a few builds in my builds/ dir now.
Let's just keep the last 3 builds around? Any reason to keep less/more?

The ostree and rpm-ostree projects have a robust set of tests that could be used to validate functionality on RHCOS/FCOS. Specifically, the vmcheck tests of rpm-ostree are such that they can be run against a VM which has an rpm-ostree daemon running.

While experimenting running these tests against RHCOS (without using kola), I ran into a few snags that made me question how the integration of the tests into kola might work.

1. The upstream tests can change with each release/commit, so kola would have to perform a checkout of the repo which matches the version of software under test. I didn't find any other examples of this kind of approach in the current codebase. Obviously, this doesn't mean we can't adapt, but I wanted to get a feel if this kind of approach would work out.

2. I found the vmcheck suite has a decent amount of dependencies that need to be installed on the host running the test. kola doesn't really have this problem, as it is just a golang binary after it is all compiled etc. Would this requirement on certain packages get pushed up to a higher layer to handle?

3. The alternative to installing packages on the host running the test would be to pull a container with the required dependencies and execute the tests via the container, via kola. I'm not sure this is a better solution than what would be required for point 2.

4. The extreme alternative is to punt on the upstream tests and re-implement some of the tests natively in 'kola`.

Maybe I'm getting ahead of myself with these kinds of questions, but I figured the folks more experienced with this codebase would be able to steer me in the right direction.

I've been developing a proof of concept bare metal installer piggybacking Anaconda kickstart functionality.
It doesn't use almost any of the "core" Anaconda features, but it seems to do what it has to do.
Please have a look, @cgwalters and others.

#Kickstart
cmdline

lang en_US.UTF-8
keyboard us
timezone --utc America/New_York
####################################################
# Not used! Fool anaconda into thinking we actually
# set a root password.
rootpw  arandomone
#####################################################

####################################################
# This is again kind of useless, as we are going to
# dd the sys image anyway later.
####################################################
clearpart --all --drives=notexistent
autopart --nolvm

network --onboot yes --device ens3 --bootproto dhcp --noipv6

%pre --erroronfail --interpreter /bin/bash --log /tmp/ks.log
#!/bin/bash
##########################################################
# Defaults - edit these to tune the installation
##########################################################
IMG_SERVER_ADDRESS=192.168.1.xxx
DEST_DEV=/dev/sda
IMAGE_URL="http://${IMG_SERVER_ADDRESS}/os-img.img.gz"
#SIG_URL="http://${IMG_SERVER_ADDRESS}/os-img.img.gz.sha256sum"

###########################################################
# Hacks galore - kill shell window and show (our) logs
###########################################################
tmux kill-window -t anaconda:shell
tmux kill-window -t anaconda:log

touch /tmp/install.log

tmux new-window -d -n log "tail -F /tmp/install.log"
tmux select-window -t anaconda:log

echo "Installing System Image" >> /tmp/install.log
echo "-----------------------------------" >> /tmp/install.log

############################################################
#Get the image info
############################################################
let retry=0
while true
do
	echo "Getting image info from $IMAGE_URL" >> /tmp/install.log

	curl -sIf $IMAGE_URL >/tmp/image_info 2>&1
	RETCODE=$?
	if [ $RETCODE -ne 0 ]
	then
		if [ $RETCODE -eq 22 -a $retry -lt 5 ]
		then
			# Network isn't up yet, sleep for a sec and retry
			sleep 1
			let retry=$retry+1
			continue
		fi
		exit 1
	else
		IMAGE_SIZE=$(cat /tmp/image_info | awk '/.*Content-Length.*/ {print $2}' | tr -d $'\r')
		TMPFS_MBSIZE=$(($IMAGE_SIZE/(1024*1024)+50))
		echo "Image size is $IMAGE_SIZE" >> /tmp/debug
		echo "tmpfs sized to $TMPFS_MBSIZE MB" >> /tmp/debug
		break;
	fi
	rm -f /tmp/image_url
done

#########################################################
#Create the tmpfs filesystem to store the image
#########################################################
echo "Mounting tmpfs" >> /tmp/debug
mkdir -p /mnt/dl
mount -t tmpfs -o size=${TMPFS_MBSIZE}m tmpfs /mnt/dl

############################################################
#Figure out the signature file type
############################################################
echo "Getting SIG_URL $SIG_URL" >> /tmp/debug
curl -sIf $SIG_URL > /dev/null 2>&1
RETCODE=$?
if [ $RETCODE -ne 0 ]
then
	echo "$SIG_URL not found" >> /tmp/debug
	SIG_URL=$IMAGE_URL.sha256sum
	echo "Getting SIG_URL $SIG_URL" >> /tmp/debug
	curl -sI $SIG_URL > /dev/null 2>&1
	if [ $? -ne 0 ]
	then
		SIG_TYPE=none
	else
		SIG_TYPE=sha
	fi
else
	SIG_TYPE=gpg
fi

echo "SIGNATURE TYPE IS $SIG_TYPE" >> /tmp/debug

#########################################################
#And Get the Image
#########################################################
echo "Downloading install image" >> /tmp/install.log

curl -s -o /mnt/dl/imagefile.gz $IMAGE_URL
if [ $? -ne 0 ]
then
	exit 1
fi

#########################################################
#Get the corresponding signature file
#########################################################
echo "Getting signature $SIG_URL" >> /tmp/install.log
curl -s -o /mnt/dl/imagefile.gz.sig $SIG_URL
if [ $? -ne 0 ]
then
	exit 1
fi

#########################################################
#Validate the integrity of the image
#########################################################
if [ "$SIG_TYPE" != "none" ]
then
	if [ "$SIG_TYPE" == "gpg" ]
	then
		gpg --trusted-key "${GPG_LONG_ID}" --verify /mnt/dl/imagefile.gz.sig >/dev/null 2>&1
		if [ $? -ne 0 ]
		then
			exit 1
		fi
	elif [ "$SIG_TYPE" == "sha" ]
	then
		sed -i -e"s/$/\ \/mnt\/dl\/imagefile\.gz/" /mnt/dl/imagefile.gz.sig
		sha256sum -c /mnt/dl/imagefile.gz.sig
		if [ $? -ne 0 ]
		then
			exit 1
		fi
	else
		sleep 3
	fi
else
	sleep 3
fi

#########################################################
#Wipe any remaining disk labels
#########################################################
dd conv=nocreat count=1024 if=/dev/zero of="${DEST_DEV}" \
        seek=$(($(blockdev --getsz "${DEST_DEV}") - 1024)) status=none

#########################################################
#And Write the image to disk
#########################################################
echo "Writing disk image" >> /tmp/install.log
gzip -dc /mnt/dl/imagefile.gz | dd conv=nocreat of="${DEST_DEV}" bs=1M status=none
udevadm settle

echo "Install complete. Rebooting in 3 seconds..." >> /tmp/install.log
sleep 3
shutdown -r now

%end

Currently the container is built from Fedora userspace, and Fedora hosts have 0666 for /dev/kvm. Other systems may not.

The easiest fix here is probably to make doubly sure we have our own copy of /dev (both podman and docker do this I believe), then rm -f /dev/kvm; mknod /dev/kvm && setfacl -m u:$USER:rw /dev/kvm or so.

I think we should have this container come opinionated out of the box with everything needed to generate a Fedora CoreOS like system. We shouldn't include the installer ISO or RPMs - but we should have a single command that makes it easy to mirror those, and after that the system can run "offline".

Now, the next big topic: anyone who wants to make a custom thing derives from the Fedora CoreOS defaults (by default).

This means that e.g. the equivalent of https://pagure.io/fedora-atomic should get "baked" into this container. (It doesn't mean that the config needs to live in this repo itself - but we should bake it in)

We should add a new toplevel config file that makes it trivial to override a few things:

• The naming
• Add/customize/remove packages
• The partitioning
• The update URL

And on the last point, I'd also like us to be opinionated about one public cloud way (e.g. S3) to publish content out of the gate, and also support "plain directory" (NFS/whatever).

Major bonus: We make this container trivial to run on a Fedora CoreOS VM in GCE with nested virt enabled.

I was testing changes to the input repos on an RHCOS build and tried to switch git repos using init --force, but found that init didn't try to pull the new repo.

I know that --force is there to ignore non-empty directories, but should it's meaning be extended to include more than that?

If not, would another flag be useful for this kind of "big hammer" operation where everything in src is replaced?

While exploring options for implementing a more complex RHCOS version string (openshift/os#360), I thought about extending the buildprep stage to be able to run scripts as part of the stage. These could be thought of like "plugins" to the buildprep stage.

I envisioned a buildprep directory (maybe it should be plugins?) that would live alongside the the manifest.yaml for a project. Inside the directory could be executable scripts that would perform additional actions to prep the directory for building.

For example, in RHCOS we need to do additional steps to generate repo files before we can do an actual build. Instead of requiring the build system or user to do those steps, buildprep could read the scripts in the directory and do those steps as part of the stage.

I think this idea would also change how build would operate, in that we would probably want to run buildprep implicitly when doing build.

Does this idea seem useful?

Should we disallow init unless we are in an empty directory? Several times I've run init when I was in the wrong directory and it has littered files in places I didn't want them. We could

1. require an emty directory
2. if running init more than once is desirable then we can just place a file in the SDK directory that let's us know this is an SDK directory.

We should probably make all 'init' functions idempotent so it can be run multiple times. Ansible comes to mind.

We've been abbreviating coreos-assembler as COSA in conversations more recently. From the beginning of the project I've also been using an alias alias cass=... to shorten the keystrokes and to allow the alias to handle the complex CLI that needs to be run for the container runtime. Since COSA has taken on some traction recently (and I like it better than cass) I propose that we formalize the abbreviation of coreo-assembler to cosa in some of our docs and within the container itself.

Proposal:

• use cosa in README and examples in docs
• add cosa -> coreos-assembler symlink inside of container
  • for symmetry for people who run from inside the container
• Don't rename the project
  • I believe the abbreviation is easily tied back to the full project name

All of these points are up for debate, including if we want to bikeshed on the cosa name.

WDYT?

I've been developing a proof of concept Bare metal installer making use of the dracut initrd building system, by creating a dracut module specifically created to run a modified version of the coreos-installer script (available here https://github.com/nhorman/coreos-dracut ). Compared to the @jlelli POC here (#228), this solution makes no use of anaconda, and is therefore smaller in size and complexity, but also is unable to take advantage of any of the features offered in anaconda, should that be desireable in the future.

to build an initrd that is capable of installing coreos:

1. clone the git repository https://github.com/nhorman/coreos-dracut.git
2. copy the contents of the dracut-module directory to /usr/share/dracut/modules/90coreos
3. In the build directory of the repository, run "make x86_64" (noting that the kernel version can be specified with the KVER environment variable). this will provide the following build outputs:
   a) an initrd image
   b) A kernel vmlinuz image
   c) a coreos.iso file
4. Item (c) can be used to burn to a dvd/cd and booted to run the installer, while items (a) and (b) are suitable for pxe booting systems over a network

Using either the pxe or cdrom method of boot, makes the following kernel command line options available:
coreos.install_dev - The block device on the system to install to
coreos.image_url - The url of the coreos image to install to this device
coreos.ignition_url - The url of the coreos ignition config (optional, enter coreos.ignition_url=skip to not load an ignition config)

If the install_dev and image_url options are specified, the installer will run autonomously. If they are not specified, the installer will become interactive, prompting the user for input on the console

This is the error I see when trying to run this on the ppc64le architecture.

[root@rhel-ocpapp1 coreos]# coreos-assembler init https://github.com/coreos/fedora-coreos-config
Trying to pull quay.io/cgwalters/coreos-assembler...Getting image source signatures
Copying blob sha256:e69e955c514f72cef9e9e7db408266a32ba48133f0e2a27d5e4b52ed1995d864
 85.99 MB / 85.99 MB [=====================================================] 10s
Copying blob sha256:a8a4d821e15658a276717ed65d865469e12dabbde7cf1871c755025118b58f3b
 160 B / 160 B [============================================================] 0s
Copying blob sha256:99a83b419d3cb4c515d32e0af4ca354f48dd4fdc40872f7bd0e4b939a04afee8
 2.53 KB / 2.53 KB [========================================================] 0s
Copying blob sha256:8faffdba319aeddefcd0a7598774221bf6ac3e7ba136bc147e0231a74e6166fc
 1.16 KB / 1.16 KB [========================================================] 0s
Copying blob sha256:a417b5e014d6bfb6b81ce057971d82f67739a0c9ebd47dd615939a63255cd7a7
 561.86 MB / 561.86 MB [===================================================] 56s
Copying blob sha256:acf8382988a511a2e77d66b6db5d6ab926198625b1d451a21040458b984184ba
 3.51 MB / 3.51 MB [========================================================] 0s
Copying blob sha256:b4118eadddddafb8e15b73ac929c6cb0e40a53036d49d87c611982c9c66adf96
 169.87 MB / 169.87 MB [===================================================] 19s
Copying blob sha256:134fd40227f49150593822289e2fd1efdf359ec2730aaf6ab01488001eab3b65
 3.32 KB / 3.32 KB [========================================================] 0s
Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
 32 B / 32 B [==============================================================] 0s
Copying blob sha256:710caf03ad410b4ffd7ae487eb56e51e6a3b58f16a9711f551f03cbed8a40973
 137 B / 137 B [============================================================] 0s
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Writing manifest to image destination
Storing signatures
standard_init_linux.go:190: exec user process caused "exec format error"

now that f29 is out it should be safe to switch to it for our assembler container

We have an outstanding discussion about using the plume bits from mantle, or using the /usr/bin/aws CLI, etc.

I personally find shell-scripting aws to not be too bad, see e.g.: openshift/os#330

The previous RHCOS build pipeline extracted the pkglist for each build which we found convenient. Let's go the next step and extract the whole rpm-ostree commit metadata as commitmeta.json or so.

The primary use case here is getting the full package list.

The mantle* artifacts are the only ones we don't pull from somewhere else. Building them as part of the container build means that we have to download build deps, compile, remove build deps, etc..

It would be better if we could grab the mantle* software pre-built rather than have to do this.

note this mostly came up when discussing how to better leverage podman build --layers for layer caching while hacking on the coreos-assembler container sources.

The tool tells you to use --force-nocache to ignore the cache while building, but this appears not to work.

No apparent changes since previous commit; use --force-nocache to override
No changes in image inputs.
[root@fedora-1 coreos]# coreos-assembler build --force-nocache
getopt: unrecognized option '--force-nocache'
Usage: coreos-assembler build --help
       coreos-assembler build [--force] [--skip-prune]

  Build OSTree and image artifacts from previously fetched packages.
[root@fedora-1 coreos]# coreos-assembler build --force

Currently the model is that builds.json implements a linear stream of builds, a lot like ostree/git commits. But we should add the concept of tagging.

See prior discussion in e.g. #177

What I'm thinking here is that builds.json would look like:

{ "builds": ["42.4", "42.3", "42.2"]
  "tags": [{"name": "smoketested",
               "created": "2018-09-26T15:51:42Z",
               "target": "42.3"}, ...]
}

Created by e.g.: coreos-assembler build-tag create smoketested 42.3

Our pruning logic would be enhanced to understand these additional references.

I'm looking at implementing the GPG signing of artifacts from coreos-assembler as requested in openshift/os#362. (I'll start with the metadata files, but extending to all artifacts should be possible. Before writing a bunch of code in the wrong direction, I wanted to get feedback from folks.

Should this kind of functionality be a separate command? For example:

$ coreos-assembler sign --help
Usage: coreos-assembler sign --help
       coreos-assembler sign [--all] [--metadata] [--images] GPGKEY 

  Sign artifacts with GPGKEY.  Defaults to signing all artifacts.  Can sign only
  metadata files or image files.

Or maybe give each stage that produces a --sign option? For example:

$ coreos-assembler build --help
Usage: coreos-assembler build --help
       coreos-assembler build [--force] [--skip-prune] [--sign] [--gpgkey GPGKEY]

  Build OSTree and image artifacts from previously fetched packages. May optionally
  GPG sign metadata files and images using --sign and providing a GPGKEY.

My preference would be the former option, as it would appear to be the last thing you would want to do before publishing any of the artifacts generated by coreos-assembler. And it would likely be easier to implement in one place, rather than all over the place.

Thoughts?

Let's have a default model where a developer or test process can do:

# mkdir -p local-overrides
# cp /path/to/kernel.rpm local-overrides

And if we find any RPMs in there we automatically run createrepo_c and add it to the config. One thing this will then hit is: rpm-software-management/libdnf#149
Which...hacking around would be very painful.

We could wrap the existing https://github.com/projectatomic/rpmdistro-gitoverlay that we ship, or do something else (more gentoo like?)

Integrate this so it's still just coreos-assembler build.

We started talking about this in #84, and it seems like we should decide on this soon-ish since the assembler is constantly growing and the longer we wait, the harder the rewrite will be.

To summarize, it seemed like the two likely candidates left were Python and Go. One point brought up in favour of Python was verbose error handling in Go, and another in favour of Go was static type checking.