build: install imagefactory? about coreos-assembler HOT 9 CLOSED

OK, I got virt-install to do what we want here (combining the above-linked portal doc with tweaks from this page to set the console args), and I also added cache=unsafe,backing_format=qcow2 to speedily get a qcow2.

virt-install --connect qemu:///session --name atomic-inst --memory 8192 --vcpus 4 \
--disk size=50,cache=unsafe,backing_format=qcow2 --location \
/srv/iso/Fedora-AtomicHost-ostree-x86_64-28-20180709.0.iso --os-variant rhel7 \
--initrd-inject /srv/walters/tmp/atomic.ks \
--extra-args="ks=file:/atomic.ks console=tty0 console=ttyS0,115200n8" --rng /dev/urandom \
--wait 10 --noreboot --nographics

This all works fine for me in a container run via podman run --security-opt seccomp=unconfined --rm -ti --device /dev/kvm ....

from coreos-assembler.

It's possible to run imagefactory in a container. It's a bit trickier to set up because it pulls in systemd + libvirt etc, which still requires some finagling to get right. I normally just use a traditional (non-Atomic) cloud VM with nested virt and just dnf install -y imagefactory-plugins-TinMan.

For the RHCOS pipeline specifically, we don't currently run imagefactory in this container, but directly on a traditional VM with imgfac installed.

from coreos-assembler.

Yeah, converting to use ImageFactory inside a container is a prereq for adding it here (conceptually). It's #5 in the releng ideas issue.

from coreos-assembler.

I tried to go one step further from here and containerize imagefactory. Even with the setup described above (a privileged Docker container, "booting" into a full systemd systemd session, with the libvirtd service running and the KVM bind-mounted from the host) this still fails because libvirt does not really like the cgroup hierarchy. VM spawning fails on libvirtd side with:

error : virFileReadAll:1420 : Failed to open file '/sys/fs/cgroup/cpuset/<CID>/cpuset.cpus': No such file or directory

from coreos-assembler.

I was playing around with doing

[libvirt]
uri = qemu:///session

in /etc/oz/oz.cfg for this but for some reason oz wasn't using it. That's the path that makes the most sense IMO to pursue, rather than trying to have the "full libvirt" daemon work inside a container.

from coreos-assembler.

In the end...the oz/imagefactory thing is very unwieldy for what we need. We could simply try using virt-install --connect qemu://session - it's basically this. Then extract the VM-provider specific code out of ImageFactory into standalone executables that can run on an existing disk image.

from coreos-assembler.

I kind of agree. it would be nice if we could document our needs and why imagefactory doesn't work for us so we can understand what changes we would like to make.

from coreos-assembler.

VM spawning fails on libvirtd side with:

I don't entirely remember the details, but I've worked around this in the past by mounting -v /sys/fs/cgroup:/sys/fs/cgroup.

from coreos-assembler.

Closing in favor of https://github.com/cgwalters/coreos-assembler/pull/9.

from coreos-assembler.