PMU causes guestfish to fail on VMware hypervisors about coreos-assembler HOT 10 CLOSED

are you building locally? I know @miabbott was hitting some issues with the VMs, but I think his were way earlier than this. Any chance your machine ran out of disk or memory ?

from coreos-assembler.

Can you do a coreos-assembler shell, and inside there do e.g. env LIBGUESTFS_DEBUG=1 coreos-assembler gf-oemid /path/to/qemu.qcow2.

from coreos-assembler.

@dustymabe the VM has 32 GB of memory, I've got more in the hypervisor, but I'd be surprised if I'm hitting that. As for disk, I'm got 17 GB free after the qcow2 image is already built.

@cgwalters that command failed.

[coreos-assembler]$ export LIBGUESTFS_DEBUG=1
[coreos-assembler]$ coreos-assembler gf-oemid /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
Unknown command: gf-oemid

from coreos-assembler.

Sorry, it should be env LIBGUESTFS_DEBUG=1 /usr/lib/coreos-assembler/gf-oemid /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2 /srv/tmp/fcos-qemu.qcow2 qemu

from coreos-assembler.

Okay, so it does appear to be a VMware hypervisor issue with nested VMs. gotoz/runq#2 looks like possibly the same problem. They have a fix by passing some additional CPU args to qemu in gotoz/runq@ee66e5a.

It's not clear to me if this has to be tried in coreos-assembler, supermin, or guestfish. I'll give it some effort and see if I can make it work though.

coreos-assembler_gf-oemid.log

from coreos-assembler.

If you see your original log the anaconda install (which does run in a VM) did work, but that is run via virt-install and thus probably calls qemu-kvm with different options than guestfish.

From your last log these are interesting statements:

qemu-system-x86_64: error: failed to set MSR 0x38d to 0x0
qemu-system-x86_64: /builddir/build/BUILD/qemu-2.11.2/target/i386/kvm.c:1807: kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

Try running the virt-install command from your original log in the container and add --dry-run --print-xml and see if there is anything interesting in the xml for cpu information. The guestfish command that is failing is using -cpu host.

from coreos-assembler.

http://libguestfs.org/guestfs.3.html#qemu-wrappers

from coreos-assembler.

I ran

[coreos-assembler]$ qemu-img create -f qcow2 /srv/tmp/build/tmp/fedora-coreos-29-base-2.qcow2 8G
[coreos-assembler]$ virt-install --connect=qemu:///session --name=coreos-inst-13281-1541708728 --noautoconsole --console=log.file=/srv/tmp/build/install.log --filesystem=source=/srv/tmp/build/tmp/anaconda,target=/mnt/logs,accessmode=mapped --filesystem=source=/srv/repo,target=/mnt/ostree-repo,accessmode=mapped --wait=10 --noreboot --nographics --memory=2048 --vcpus=sockets=4,cores=1,threads=1 --os-variant=rhel7 --rng=/dev/urandom --check path_in_use=off --network=none --disk=path=/srv/tmp/build/tmp/fedora-coreos-29-base-2.qcow2,cache=unsafe --location=/srv/installer/Fedora-Everything-netinst-x86_64-28-1.1.iso --initrd-inject=/srv/tmp/build/tmp/flattened.ks --extra-args "ks=file://flattened.ks console=tty0 console=ttyS0,115200n8 inst.cmdline inst.notmux" --dry-run --print-xml

It produced the attached xml. I'm working on the qemu-wrapper shim. I'll report back the results.

dry_run.txt

from coreos-assembler.

Sweet. Looks like it worked. If it works on non-vmware hypervisors can we just enable the CPU flag using the wrapper across the board?

[coreos-assembler]$ export  LIBGUESTFS_HV=$(pwd)/qemu-wrapper.sh
[coreos-assembler]$ export LIBGUESTFS_DEBUG=1
[coreos-assembler]$ cat qemu-wrapper.sh
#!/bin/bash -

i=0
while [ $# -gt 0 ]; do
    case "$1" in
    -cpu)
        shift 2;;
    *)
        args[i]="$1"
        (( i++ ))
        shift ;;
    esac
done

exec qemu-kvm -cpu host,pmu=off "${args[@]}"
[coreos-assembler]$ /usr/lib/coreos-assembler/gf-oemid /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2 /srv/tmp/fcos-qemu.qcow2 qemu

guestfish_success.log

from coreos-assembler.

FYI #465 drops this as I think it may be causing problems for us elsewhere. If we need to we can revisit adding this though.

from coreos-assembler.