coreos-assembler breaks on ppc64le about coreos-assembler HOT 74 CLOSED

Yeah, probably worth doing a release this week to reduce friction for coreos-assembler users, though I'll mention that you should be able to use the rpm-ostree from FAHC for the time being.

from coreos-assembler.

That could be a new Koji tag, it could be something like rpmdistro-gitoverlay running on multiple architectures, it could be ensuring COPR has arch parity - whichever.

Migrating this particular discussion to coreos/fedora-coreos-tracker#84 since I think it warrants it's own high level topic.

from coreos-assembler.

For the purposes of the exercise for now, I'd recommend just building rpm-ostree from source in the container. I can make a ppc64le scratch build in Koji if you'd like? Let me do that.

from coreos-assembler.

It turns out that SMBIOS is only available on x86 and ARM. fw_cfg is available on more architectures, but not ppc64:

https://github.com/torvalds/linux/blob/ef78e5ec9214376c5cb989f5da70b02d0c117b66/drivers/firmware/Kconfig#L199-L201

Well, we don't really need qemu_fw_cfg. It just makes it really convenient to test locally, right?

At present, if we don't have it, there's no reasonable way to pass an Ignition config to a ppc64 QEMU VM (coreos/ignition#666). (It'd have to be done by netbooting the guest or modifying the disk image.) That would also affect production virtualization.

Otherwise I guess we could recommend config drives or coreos.config.url for local testing on those platforms?

By policy, Ignition doesn't support config drives.

from coreos-assembler.

hi @manojnkumar. Thanks for trying out coreos-assembler!

This is probably not going to work out of the box right now because we haven't done any enablement work to get it to happen yet. One example would be the container runtime is pulling an x86_64 container from quay.io (we haven't built it for non x86 architectures yet).

In order to get this to work you'll need to build the container locally on ppc64le hardware first and then use the container you just built as the assembler container instead of quay.io/cgwalters/coreos-assembler. Does that make sense?

You'll probably hit other issues along the way, but let's document them and we can try to get it working.

from coreos-assembler.

I cloned this project and ran the coreos-assembler script directly and running into the following pre-requisites:

/home/cloud-user//coreos-assembler/coreos-assembler init https://github.com/coreos/fedora-coreos-config
...
error: Failed to find expected dependencies: rpm-ostree dnf-utils fedpkg rpmdistro-gitoverlay distribution-gpg-keys python2-gobject-base python3-gobject-base podman buildah jq awscli ignition python3-dateutil

@dustymabe : Could you provide me instructions on building a ppc64le container?

from coreos-assembler.

Should I be building on RHEL, or somewhere else (CentOS etc.)?

from coreos-assembler.

I cloned this project and ran the coreos-assembler script directly

the tool primarily expects to be run from within a container. So you'd need to clone this repo and then do something like podman build -t coreos-assembler . (you can use docker there if you prefer). Then set up your alias to use coreos-assembler rather than quay.io/cgwalters/coreos-assembler and run it again.

Does that make sense?

Should I be building on RHEL, or somewhere else (CentOS etc.)?

I run on Fedora usually, but it shouldn't matter that much I don't think because the environment is in the container.

from coreos-assembler.

docker build fails at this stage:

=False

• read line
• mv /etc/yum.repos.d/fedora.repo.new /etc/yum.repos.d/fedora.repo
• cat
• cat
  ---> fe56827f4074
  Removing intermediate container ca31db715dba
  Step 5/12 : RUN ./build.sh install_rpms
  ---> Running in 72e39808c1a3

++ pwd

• srcdir=/root/containerbuild
• install_rpms
• dnf -y distro-sync
  Repository 'fahc' is missing name in configuration, using id.
  Error: Failed to synchronize cache for repo 'updates'
  The command '/bin/sh -c ./build.sh install_rpms' returned a non-zero code: 1

from coreos-assembler.

* Error: Failed to synchronize cache for repo 'updates'

That's typically a network intermittent failure. If you run it multiple times does it continue to die in the same place?

cc @sinnykumari who might be able to try this on some ppc64le hardware she has.

from coreos-assembler.

I tried this on several places, where I kept seeing similar errors. Finally I hit a server where I was able to go further:

Installed:
findutils.ppc64le 1:4.6.0-19.fc28

Complete!
++ grep -v '^#' /root/containerbuild/build-deps.txt

• self_builddeps=golang
• echo golang
• grep -v '^#' /root/containerbuild/deps.txt
• xargs dnf -y install
  Repository 'fahc' is missing name in configuration, using id.
  Failed to synchronize cache for repo 'dustymabe-ignition', disabling.
  Failed to synchronize cache for repo 'walters-buildtools-fedora', disabling.
  Last metadata expiration check: 0:00:00 ago on Wed Nov 7 19:05:26 2018.
  No match for argument: rpmdistro-gitoverlay
  Package python3-gobject-base-3.28.3-1.fc28.ppc64le is already installed, skipping.
  Error: Unable to find a match
  The command '/bin/sh -c ./build.sh install_rpms' returned a non-zero code: 123

from coreos-assembler.

findutils.ppc64le 1:4.6.0-19.fc28

hmm. this seems odd. Are you trying to build an f28 image? We switched to f29 some time ago.

* Failed to synchronize cache for repo 'dustymabe-ignition', disabling.

There aren't any ppc64le builds for this for f28, there is for f29

  Failed to synchronize cache for repo 'walters-buildtools-fedora', disabling.

@cgwalters can you enable ppc64le for your copr repo? https://copr.fedorainfracloud.org/coprs/walters/buildtools-fedora/

from coreos-assembler.

findutils.ppc64le 1:4.6.0-19.fc28

hmm. this seems odd. Are you trying to build an f28 image? We switched to f29 some time ago.

actually. i was mistaken.. this is the container build itself. I have opened #195 for us to switch to f29 for the assembler container.

from coreos-assembler.

For now, once colin has enabled ppc64le builds for his copr (wait to hear back from him here in this issue), you can switch to f29 in the Dockerfile and try again.

from coreos-assembler.

Switching to f29 in the Dockerfile, hit the same issue:

Complete!

• dnf -y install /usr/bin/xargs
  Repository 'fahc' is missing name in configuration, using id.
  fahc 4.5 kB/s | 1.5 kB 00:00
  Copr repo for buildtools-fedora owned by walter 980 B/s | 341 B 00:00
  Failed to synchronize cache for repo 'walters-buildtools-fedora', ignoring this repo.
  Package findutils-1:4.6.0-21.fc29.ppc64le is already installed.
  Dependencies resolved.
  Nothing to do.
  Complete!
  ++ grep -v '^#' /root/containerbuild/build-deps.txt
• self_builddeps=golang
• echo golang
• grep -v '^#' /root/containerbuild/deps.txt
• xargs dnf -y install
  Repository 'fahc' is missing name in configuration, using id.
  Copr repo for buildtools-fedora owned by walter 1.1 kB/s | 341 B 00:00
  Failed to synchronize cache for repo 'walters-buildtools-fedora', ignoring this repo.
  Last metadata expiration check: 0:00:04 ago on Wed Nov 7 21:22:02 2018.
  No match for argument: rpmdistro-gitoverlay
  Error: Unable to find a match
  The command '/bin/sh -c ./build.sh install_rpms' returned a non-zero code: 123

from coreos-assembler.

This is the same issue as #30 (comment) which was for aarch64.

Although, since COPR does support ppc64le I added that to the buildroots and did the "update revision" dance and started a new build: https://copr.fedorainfracloud.org/coprs/walters/buildtools-fedora/build/820725/

Like I said in that ticket though...it may be simplest to drop rdgo from this container for now.

from coreos-assembler.

now that colin has enabled ppc64le and the build succeeded, you can try it again now and see where it breaks this time :)

from coreos-assembler.

Thanks @cgwalters @dustymabe . It now fails here:

Package findutils-1:4.6.0-21.fc29.ppc64le is already installed.
Dependencies resolved.
Nothing to do.
Complete!
++ grep -v '^#' /root/containerbuild/build-deps.txt

• self_builddeps=golang
• echo golang
• grep -v '^#' /root/containerbuild/deps.txt
• xargs dnf -y install
  Repository 'fahc' is missing name in configuration, using id.
  Last metadata expiration check: 0:00:03 ago on Wed Nov 7 22:22:28 2018.
  Error:
  Problem 1: conflicting requests
  • package rpm-ostree-2018.9.38-2c4231b3769a99af36fdd68d03c1b24468f3f5b2.d8a5bf5d7acae1bdd57001f2a39aa8e890f79c0f.fc28.x86_64 does not have a compatible architecture
  • nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by rpm-ostree-2018.9.38-2c4231b3769a99af36fdd68d03c1b24468f3f5b2.d8a5bf5d7acae1bdd57001f2a39aa8e890f79c0f.fc28.x86_64
  • nothing provides rpm-ostree-libs(x86-64) = 2018.9.38-2c4231b3769a99af36fdd68d03c1b24468f3f5b2.d8a5bf5d7acae1bdd57001f2a39aa8e890f79c0f.fc28 needed by rpm-ostree-2018.9.38-2c4231b3769a99af36fdd68d03c1b24468f3f5b2.d8a5bf5d7acae1bdd57001f2a39aa8e890f79c0f.fc28.x86_64
    Problem 2: package skopeo-1:0.1.32-2.dev.gite814f96.fc29.ppc64le requires libostree-1.so.1()(64bit), but none of the providers can be installed
  • package skopeo-1:0.1.32-2.dev.gite814f96.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.3)(64bit), but none of the providers can be installed
  • package skopeo-1:0.1.32-2.dev.gite814f96.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.8)(64bit), but none of the providers can be installed
  • conflicting requests
  • package ostree-libs-2018.8-1.fc29.ppc64le is excluded
    Problem 3: package buildah-1.4-1.dev.git0a7389c.fc29.ppc64le requires libostree-1.so.1()(64bit), but none of the providers can be installed
  • package buildah-1.4-1.dev.git0a7389c.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.3)(64bit), but none of the providers can be installed
  • package buildah-1.4-1.dev.git0a7389c.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.8)(64bit), but none of the providers can be installed
  • conflicting requests
  • package ostree-libs-2018.8-1.fc29.ppc64le is excluded
    Problem 4: conflicting requests
  • package podman-1:0.10.1.3-4.gitdb08685.fc29.ppc64le requires libostree-1.so.1()(64bit), but none of the providers can be installed
  • package podman-1:0.10.1.3-4.gitdb08685.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.3)(64bit), but none of the providers can be installed
  • package podman-1:0.10.1.3-4.gitdb08685.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.8)(64bit), but none of the providers can be installed
  • package podman-1:0.10.1-1.gite4a1553.fc29.ppc64le requires libostree-1.so.1()(64bit), but none of the providers can be installed
  • package podman-1:0.10.1-1.gite4a1553.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.3)(64bit), but none of the providers can be installed
  • package podman-1:0.10.1-1.gite4a1553.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.8)(64bit), but none of the providers can be installed
  • package ostree-libs-2018.8-1.fc29.ppc64le is excluded
    The command '/bin/sh -c ./build.sh install_rpms' returned a non-zero code: 123

from coreos-assembler.

The fahc repo is generated by https://ci.centos.org/view/Atomic/job/fahc-rdgo/ which is indeed currently x86_64 only.

It's tricky since today c-a tends to rely on rpm-ostree built from git master. This all loops back to sadly Koji (and to a lesser degree COPR) "owning" the multi-arch hardware; makes it harder for other projects that want to do CI or builds differently to use it too. This is https://github.com/projectatomic/rpmdistro-gitoverlay/blob/master/doc/reworking-fedora-releng.md#blend-upstream-testing-and-downstream-testing

Anyways so...you could probably remove the bits at the top that enable fahc, or we could switch to COPR I guess, although that takes us back to manual integration...

from coreos-assembler.

@cgwalters: Commenting out fahc does get the build much further. It now fails at:

Step 7/13 : RUN ./build.sh make_and_makeinstall
 ---> Running in 62f640557d8c

++ pwd
+ srcdir=/root/containerbuild
+ make_and_makeinstall
+ test -d .git
+ mkdir -p /usr/app/
+ rsync -rlv /root/containerbuild/ostree-releng-scripts/ /usr/app/ostree-releng-scripts/
sending incremental file list
created directory /usr/app/ostree-releng-scripts
./

sent 39 bytes  received 72 bytes  222.00 bytes/sec
total size is 0  speedup is 0.00
+ test -f mantle/README.md
+ echo 'Run: git submodule update --init'
Run: git submodule update --init
+ exit 1
The command '/bin/sh -c ./build.sh make_and_makeinstall' returned a non-zero code: 1

from coreos-assembler.

OK, finally I got the docker build to complete. I had to add the git submodule update --init to the Dockerfile. Here are my changes so far:

[root@rhel-ocpinfra1 coreos-assembler]# git diff
diff --git a/Dockerfile b/Dockerfile
index 0599f8c..be5c830 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,4 @@
-FROM registry.fedoraproject.org/fedora:28
+FROM registry.fedoraproject.org/fedora:29
 WORKDIR /root/containerbuild
 
 
 # Only need a few of our scripts for the first few steps
@@ -8,6 +8,7 @@ RUN ./build.sh install_rpms
 
 # Ok copy in the rest of them for the next few steps
 COPY ./ /root/containerbuild/
+RUN git submodule update --init
 RUN ./build.sh make_and_makeinstall
 RUN ./build.sh configure_user
 
diff --git a/build.sh b/build.sh
index 64f0703..a0d0299 100755
--- a/build.sh
+++ b/build.sh
@@ -5,19 +5,6 @@ srcdir=$(pwd)
 
 configure_yum_repos() {
 
-    # Enable FAHC https://pagure.io/fedora-atomic-host-continuous
-    # so we have ostree/rpm-ostree git master for our :latest
-    # NOTE: The canonical copy of this code lives in rpm-ostree's CI:
-    # https://github.com/projectatomic/rpm-ostree/blob/d2b0e42bfce972406ac69f8e2136c98f22b85fb2/ci/build.sh#L13
-    # Please edit there first
-    echo -e '[fahc]\nmetadata_expire=1m\nbaseurl=https://ci.centos.org/artifacts/sig-atomic/fahc/rdgo/build/\ngpgcheck=0\n' > /etc/yum.repo
-    # Until we fix https://github.com/rpm-software-management/libdnf/pull/149
-    excludes='exclude=ostree ostree-libs ostree-grub2 rpm-ostree'
-    for repo in /etc/yum.repos.d/fedora*.repo; do
-        cat ${repo} | (while read line; do if echo "$line" | grep -qE -e '^enabled=1'; then echo "${excludes}"; fi; echo $line; done) > ${r
-        mv ${repo}.new ${repo}
-    done
-
     # enable `walters/buildtools-fedora` copr
        # pulled from https://copr.fedorainfracloud.org/coprs/walters/buildtools-fedora/repo/fedora-28/walters-buildtools-fedora-fedora-28.r
     cat > /etc/yum.repos.d/walters-buildtools-fedora-fedora-28.repo  <<'EOF'

from coreos-assembler.

Also looking at the logs I need to add something here for ppc64le?

• make
  cd mantle && ./build ore kola kolet
  Building ore
  Building kola
  Building amd64/kolet
  Building arm64/kolet

from coreos-assembler.

hey @manojnkumar - I edited your earlier comment to make it easier to read. Here is a guide on markdown I've been using for github: https://guides.github.com/features/mastering-markdown/

Also looking at the logs I need to add something here for ppc64le?

I don't know if we've looked at building the mantle codebase on other architectures. @bgilbert, @arithx, can you comment on that?

@manojnkumar, for now you can comment out the cd mantle && ./build ore kola kolet as it's not explicitly needed for build.

from coreos-assembler.

Thanks @dustymabe. I proceeded with the coreos-assembler init and build steps with my built container for ppc64le. The build steps fails with:

[root@rhel-ocpinfra1 coreos]# coreos-assembler build
Using manifest: /srv/src/config/manifest.yaml
bwrap: Creating new namespace failed, likely because the kernel does not support user namespaces. bwrap must be installed setuid on such systems.
error: bwrap test failed, see <https://github.com/projectatomic/rpm-ostree/pull/429>: Executing bwrap(true): Child process killed by signal 1

Any clues @cgwalters ?

from coreos-assembler.

Did you set up an alias for coreos-assembler ? If so what did you set it to? What is the OS that is on your laptop ?

from coreos-assembler.

What's your host system? Is it RHEL7? You need to enable user namespaces. See...

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/getting_started_with_containers/index#user_namespaces_options

Specifically sysctl -w user.max_user_namespaces=255888 should be all you need on RHEL (or RHELAH) 7.5+.

from coreos-assembler.

alias coreos-assembler='docker run --rm --net=host -ti --privileged --userns=host -v $(pwd):/srv --workdir /srv coreos-assembler'

Yes RHEL 7.5.

[root@rhel-ocpinfra1 coreos-assembler]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)

from coreos-assembler.

@manojnkumar Please try the sysctl command as root in #194 (comment)

It's quite safe in general, most particularly if you don't have any potentially hostile users with ssh access to the system - user namespaces are disabled by default on RHEL7 out of conservatism.

from coreos-assembler.

Doesn't seem to help @cgwalters. Have been trying to do the rest of the steps in the link you provided, and I get into a circular issue with the userns setting in /etc/sysconfig/docker and the lines in /etc/subuid

[root@manoj-ocp-service-catalog coreos]# coreos-assembler build
Using manifest: /srv/src/config/manifest.yaml
bwrap: Creating new namespace failed, likely because the kernel does not support user namespaces. bwrap must be installed setuid on such systems.
error: bwrap test failed, see coreos/rpm-ostree#429: Executing bwrap(true): Child process killed by signal 1
[root@manoj-ocp-service-catalog coreos]# sysctl -w user.max_user_namespaces=255888
user.max_user_namespaces = 255888
[root@manoj-ocp-service-catalog coreos]# coreos-assembler build
Using manifest: /srv/src/config/manifest.yaml
error: This command requires root privileges

from coreos-assembler.

I'm not sure if ore and kola currently build on non-x86, but if not, the fixes should be minor. And yes, we'd need to add a kolet build for ppc64le.

from coreos-assembler.

@cgwalters : When I try to pull a container built on another system, to a system that has userns enabled I see this error. I seem to be stuck, with now way to proceed to the build phase:

56bfe5829eb1: Download complete
0bc8291d9890: Download complete
failed to register layer: Error processing tar file(exit status 1): Container ID 1000 cannot be mapped to a host ID

from coreos-assembler.

I haven't played much with userns in Docker...can you try turning it off? To clarify, are you using the upstream Docker or the one in RHEL Extras?

from coreos-assembler.

docker is from RHEL extras @cgwalters

[root@rhel-ocpapp1 coreos]# docker --version
Docker version 1.13.1, build 6e3bb8e/1.13.1

If I turn off userns in docker, I hit the other issue:

[root@rhel-ocpapp1 coreos]# coreos-assembler build
Using manifest: /srv/src/config/manifest.yaml
error: This command requires root privileges

from coreos-assembler.

@manojnkumar - can you grab me in #fedora-coreos on IRC and we can try to work through this and get it working?

from coreos-assembler.

Hi there, I am trying also to play with coreos ppc64le and get stuck at same step (I guess).
I followed different steps describes above except I am working on a fedora 29 VM (and not a RHEL) and I am using podman instead of docker (but I will try docker too).

[root@vm90 coreos-assembler]# coreos-assembler build
Using manifest: /srv/src/config/manifest.yaml
bwrap: Unexpected capabilities but not setuid, old file caps config?
error: bwrap test failed, see coreos/rpm-ostree#429: Executing bwrap(true): Child process killed by signal 1

from coreos-assembler.

@dustymabe : Thanks for the offer. I am at the OpenStack summit this week in Berlin. So my availability, in the overlapping work period is a bit limited.

from coreos-assembler.

@manojnkumar
@dustymabe : Thanks for the offer. I am at the OpenStack summit this week in Berlin. So my availability, in the overlapping work period is a bit limited.

no worries, catch me when you get back.

@menantea
Hi there, I am trying also to play with coreos ppc64le

Hi @menantea - welcome. We'll try to get this working.

@sinnykumari, do you think you could try coreos-assembler out on ppc64le and we can document the steps and get some of the issues fixed?

from coreos-assembler.

@dustymabe Sure, I will start working on it.

from coreos-assembler.

Thanks @dustymabe , @sinnykumari , @menantea. The changes I had to make so far are up here:

#194 (comment)

from coreos-assembler.

I also commented the mantle stuff in the Makefile as it is said here: #194 (comment)

from coreos-assembler.

I gave a try to run coreos-assembler locally on ppc64le and hit issue at step coreos-assembler build .
Error message is a bit different with coreos-assembler docker and podman image.
Using podman:

$ coreos-assembler build
...
Using manifest: /srv/src/config/manifest.yaml
bwrap: Unexpected capabilities but not setuid, old file caps config?
error: bwrap test failed, see <https://github.com/projectatomic/rpm-ostree/pull/429>: Executing bwrap(true): Child process killed by signal 1

Using docker:

$ coreos-assembler build
...
Using manifest: /srv/src/config/manifest.yaml
RPM-OSTree Version: 2018.9
error: This command requires root privileges

With debugging enabled in src/cmd-build , it fails while executing rpm-ostree compose tree --repo=repo --print-only /srv/src/config/manifest.yaml in both cases.

Note that on x86_64 we are using rpm-ostree package from fahc rdgo repo built in CentOS at https://ci.centos.org/artifacts/sig-atomic/fahc/rdgo/build/rpm-ostree-2018.9.41-89486eace187ccb15545b774e26d0e39633c3f34.d8a5bf5d7acae1bdd57001f2a39aa8e890f79c0f/ , while on ppc64le we are using latest available in fedora which is rpm-ostree-2018.9-3.fc28.1.

When I tried coreos-assembler build on x86_64 with downgraded rpm-ostree to 2018.9-3.fc28.1, I see same error

bwrap: Unexpected capabilities but not setuid, old file caps config?
error: bwrap test failed, see <https://github.com/projectatomic/rpm-ostree/pull/429>: Executing bwrap(true): Child process killed by signal 1

This makes me think that we need latest rpm-ostree (from master) on other arches as well to get pass this error.
Meanwhile, I will try to figure out that which changes are causing this issue by building rpm-ostree locally.

from coreos-assembler.

This makes me think that we need latest rpm-ostree (from master) on other arches as well to get pass this error.

Yeah, #192 depends on coreos/rpm-ostree#1666 at least.

from coreos-assembler.

This makes me think that we need latest rpm-ostree (from master) on other arches as well to get pass this error.

Yeah, #192 depends on projectatomic/rpm-ostree#1666 at least.

👍 - nice work tracking that down @sinnykumari - thanks @cgwalters for spotting the issue

from coreos-assembler.

Ah, interesting. Will it be possible to cut a new rpm-ostree release soon in order to have these changes available in next fedora updates too?

from coreos-assembler.

Will it be possible to cut a new rpm-ostree release soon in order to have these changes available in next fedora updates too?

The thing is this implies a huge lag time between when we land a change in rpm-ostree and can actually use it in coreos-assembler.

What we really want is something like FAHC but multi-architecture. For now though we can probably live with enabling multiarch in COPR and doing manual builds or standing up some external automation for COPR.

from coreos-assembler.

(Moving this discussion from coreos/fedora-coreos-config#31 (comment) )

Using COPR will have implication on aarch64 unless aarch64 gets added to COPR

In the end we need a place to do builds that's faster than Koji and supports all the architectures we care about.

That could be a new Koji tag, it could be something like rpmdistro-gitoverlay running on multiple architectures, it could be ensuring COPR has arch parity - whichever.

To repeat the rationale: I really like how right now we ship git master of rpm-ostree (and ostree) in coreos-assembler. In order to do releases, we need a "tip" of people testing git master. Today rpm-ostree is one codebase that goes on both the client and server. The risk of running master on clients is a lot greater (although since we gate PRs, we do pretty well).

For example fixing #197 would probably require landing an rpm-ostree change and then a change in this repository - and that'd be a lot more painful if there was weeks of lag time between.

I guess another option is to go back to building ostree/rpm-ostree in this container but...it'd notably bloat the build dependency set.

from coreos-assembler.

(Moving this discussion from coreos/fedora-coreos-config#31 (comment) )

Using COPR will have implication on aarch64 unless aarch64 gets added to COPR

In the end we need a place to do builds that's faster than Koji and supports all the architectures we care about.

That could be a new Koji tag, it could be something like rpmdistro-gitoverlay running on multiple architectures, it could be ensuring COPR has arch parity - whichever.

koji is the first thing that came to my mind since it has resources/support to do builds for all architectures we care about.
If this is not a very complicated thing to try, I am in support for trying a new koji tag and do the periodic build.

To repeat the rationale: I really like how right now we ship git master of rpm-ostree (and ostree) in coreos-assembler. In order to do releases, we need a "tip" of people testing git master. Today rpm-ostree is one codebase that goes on both the client and server. The risk of running master on clients is a lot greater (although since we gate PRs, we do pretty well).

+1

For example fixing #197 would probably require landing an rpm-ostree change and then a change in this repository - and that'd be a lot more painful if there was weeks of lag time between.

I guess another option is to go back to building ostree/rpm-ostree in this container but...it'd notably bloat the build dependency set.

I am also not in favor of this.

from coreos-assembler.

Proceeded further for now with reverting changes from https://github.com/coreos/coreos-assembler/pull/192/files locally.

Next error during coreos-assembler build:

+ set - rpm-ostree compose tree --repo=/srv/repo --cachedir=/srv/cache --touch-if-changed /srv/tmp/treecompose.changed --unified-core /srv/src/config/manifest.yaml --cache-only --add-metadata-from-json /srv/tmp/build/tmp/commit-metadata-input.json --write-composejson-to /srv/tmp/build/tmp/compose.json
Running: rpm-ostree compose tree --repo=/srv/repo --cachedir=/srv/cache --touch-if-changed /srv/tmp/treecompose.changed --unified-core /srv/src/config/manifest.yaml --cache-only --add-metadata-from-json /srv/tmp/build/tmp/commit-metadata-input.json --write-composejson-to /srv/tmp/build/tmp/compose.json
error: --ex-unified-core requires a bare-user repository

Above error also seems to be with recent changed made in rpm-ostree coreos/rpm-ostree#1657 .
For now, I have proceeded further with creating ostree repo as bare-user at https://github.com/coreos/coreos-assembler/blob/master/src/cmd-init#L180

from coreos-assembler.

error: --ex-unified-core requires a bare-user repository

weird.. could you try to delete your working directory (/srv/ for most of us) and try again?

from coreos-assembler.

error: --ex-unified-core requires a bare-user repository

weird.. could you try to delete your working directory (/srv/ for most of us) and try again?

I have tried this after cleaning up /srv/ and then was getting the error. Note that I am using rpm-ostree-2018.9-3.fc28.1 on ppc64le. If I switch back to rpm-ostree-2018.9-3.fc28.1 on x6_64, I get same error here as well. That's why I believe that this issue should disappear on ppc64le when we start using latest rpm-ostree (coreos/rpm-ostree#1657 looks to me as one of the related change)

from coreos-assembler.

https://koji.fedoraproject.org/koji/taskinfo?taskID=31022405

from coreos-assembler.

Ahh heh, it needs a newer OSTree as well. Hmm unless... yes I can patch out that dependency. New build: https://koji.fedoraproject.org/koji/taskinfo?taskID=31022867

from coreos-assembler.

Ahh heh, it needs a newer OSTree as well. Hmm unless... yes I can patch out that dependency. New build: https://koji.fedoraproject.org/koji/taskinfo?taskID=31022867

Thanks a lot!

from coreos-assembler.

I have created repo with the latest rpm-ostree scratch build rpms which @jlebon built and is available at https://sinnykumari.fedorapeople.org/custom-build/rpm-ostree/f28/ . Will update it in future with more recent builds if needed (until we have some better solution)

from coreos-assembler.

I'm elaborating on this point here mainly because I want to be able to link to this thread later as a rationale elsewhere.

To build on/rephrase #194 (comment) a bit more:

Loosely coupled components in general are good. I think it's worked out well to have rpm-ostree separate from libostree, which is in turn separate from e.g. grub, and on a higher level an operator/ansible/gnome-software or whatever.

However: sometimes we really do want to - for a period of time - tightly couple selected components and iterate on them quickly together to solve a problem. This definitely has happened a lot with rpm-ostree/libostree, and the fact that rpm-ostree can (if we need to) depend on libostree git master in CI has been absolutely critical.

On the flip side...for example, I think the libostree/grub border is too strong, and if we had the capability to e.g. fork grub for a time and tightly couple it with libostree it would have helped solve some problems.

Same thing will happen with coreos-assembler + [orchestrating pipeline]. Ideally they are loosely coupled, but some times you just need to make coordinated changes to both.

What I'm saying here is this isn't just about coreos-assembler + rpm-ostree - it's about how we think about software delivery, packaging, and the Koji model of "components can only use other's releases" is too limiting and also discourages people from hacking on higher level or lower level parts.

from coreos-assembler.

the Koji model of "components can only use other's releases" is too limiting and also discourages people from hacking on higher level or lower level parts.

In coreos/fedora-coreos-tracker#84 I'm advocating for builds of upstream dev branches, not just releases. I think we are agreeing with one another. Am I mistaken?

from coreos-assembler.

Another issue found during coreos-asembler build during Preparing kernel kernel phase:

dracut: *** Including module: ignition ***
dracut-install: Failed to find module 'qemu_fw_cfg'
dracut: FAILED:  /usr/lib/dracut/dracut-install -D /tmp/dracut/dracut.3ymflN/initramfs --kerneldir /lib/modules/4.19.4-300.fc29.ppc64le/ -m qemu_fw_cfg
dracut: installkernel failed in module ignition
error: Finalizing rootfs: During kernel processing: Executing bwrap(rpmostree-dracut-wrapper): Child process killed by signal 1

From PR coreos/ignition-dracut#25 , it seems install of qemu_fw_cfg module was made explicit. qemu_fw_cfg module is provided by kernel-core sub-package which is available for x86_64 but not for ppc64le .

Is this module essential or we can make it on demand loading depending upon availability?

from coreos-assembler.

Is this module essential or we can make it on demand loading depending upon availability?

It's currently needed for Ignition to read userdata from QEMU; see coreos/ignition#656.

from coreos-assembler.

@sinnykumari
qemu_fw_cfg module is provided by kernel-core sub-package which is available for x86_64 but not for ppc64le

does that mean the functionality just doesn't exist on ppc64le ?

@bgilbert
It's currently needed for Ignition to read userdata from QEMU; see coreos/ignition#656.

Might be worth investigating if SMBIOS OEM string would work on ppc64le

from coreos-assembler.

@sinnykumari
qemu_fw_cfg module is provided by kernel-core sub-package which is available for x86_64 but not for ppc64le

does that mean the functionality just doesn't exist on ppc64le ?

May have to check with Fedora kernel folks. https://github.com/torvalds/linux/blob/master/drivers/firmware/qemu_fw_cfg.c#L9 talks about x86_64 and arm but mention nothing about Power.

@bgilbert
It's currently needed for Ignition to read userdata from QEMU; see coreos/ignition#656.

Might be worth investigating if SMBIOS OEM string would work on ppc64le

from coreos-assembler.

Well, we don't really need qemu_fw_cfg. It just makes it really convenient to test locally, right?

From PR coreos/ignition-dracut#25 , it seems install of qemu_fw_cfg module was made explicit.

Note this logic was tweaked in coreos/ignition-dracut#28 to support the feature baked in. See also specifically coreos/ignition-dracut#28 (comment) re. not necessarily needing it.

Might be worth investigating if SMBIOS OEM string would work on ppc64le

Ahh that'd be neat. Otherwise I guess we could recommend config drives or coreos.config.url for local testing on those platforms? (And e.g. adapt coreos-assembler run as well).

from coreos-assembler.

qemu_fw_cfg also does not exist in the kernel-core package on ARM, and AFAICT the default ARM kernel does not enable virtualization at all.

I hit the same error last week trying to build armhfp. I didn't yet report due to traveling, I wanted to dig a little deeper when back home.

from coreos-assembler.

modifying the disk image

This is cheap with qcow2 backing files (or reflinks if available). The main thing is to make it not an ergonomic hit, but I see no issue with unilaterally changing e.g. coreos-assembler run to do it. However there are other projects that use it too, like the openshift installer on libvirt.

We may end up needing to ship a script to inject ignition configs which would be used by run -and allow projects like the installer to vendor it?

from coreos-assembler.

We can ship a wrapper script (as we do in Container Linux as a convenience) but that doesn't really help users who want to run VMs directly in existing systems, e.g. libvirt. In CL we've generally felt that the right approach is to introduce proper userdata support into the various virtualization systems, rather than forcing users to go through an extra mangling step between "download VM image" and "run VM in the usual way".

In this instance, see coreos/ignition#656 (comment).

from coreos-assembler.

@sinnykumari : How far do you get with building for ppc64le? And what changes did you have to make? I am still not getting any further with the changes I had staged. Are any of your changes worth pushing?

from coreos-assembler.

@sinnykumari : How far do you get with building for ppc64le? And what changes did you have to make? I am still not getting any further with the changes I had staged. Are any of your changes worth pushing?

arch-specific changes which can be pushed upstream has already been done:

• #216
• coreos/fedora-coreos-config#31

Other than that, few changes are there which I did locally to see how far we can go with build on ppc64le, but these changes are not meant to get in officially:

• Using rpm-ostree package from repo - https://sinnykumari.fedorapeople.org/custom-build/rpm-ostree/f28/ppc64le/ during container build which can be added locally at https://github.com/coreos/coreos-assembler/blob/master/build.sh#L47
• Using custom ignition package in fedora-coreos-config which can be found at https://sinnykumari.fedorapeople.org/custom-build/ignition/f29/ppc64le/ . You can modify here https://github.com/coreos/fedora-coreos-config/blob/master/fedora-coreos-base.yaml#L17 to use the custom ignition repo before running coreos-assembler build. This is really just a workaround now to skip error described at #194 (comment) . In my custom ignition-dracut build I am forcing to not install qemu_fw_cfg module. It needs a proper solution and related issue is created at coreos/ignition#666

After doing above said changes coreos-assembler build fails during virt-install, full logs is available here .

Please note that my last attempt was around 1 week back and I didn't get time to try further(hoping to get back to it soon).

from coreos-assembler.

@sinnykumari : I am also hitting build failures. I could not find your pastebin to compare.

Also your rpm-ostree comment indicates you are trying f28, whereas the ignition comment indicates f29. I am having issues with both.

from coreos-assembler.

qemu_fw_cfg module is provided by kernel-core sub-package which is available for x86_64 but not for ppc64le

The QEMU Ignition config is a very nice to have for development but not currently a targeted production path. Production would be bare metal (PXE HTTP fetch of Ignition config or mounting the /boot partition and looking for Ignition there).

And the other production path would be OpenStack, fetching Ignition over user data just like happens on x86_64.

from coreos-assembler.

The QEMU Ignition config is a very nice to have for development but not currently a targeted production path. Production would be bare metal (PXE HTTP fetch of Ignition config or mounting the /boot partition and looking for Ignition there).

And the other production path would be OpenStack, fetching Ignition over user data just like happens on x86_64.

kola support should be a prerequisite for supporting an architecture. Running kola tests in OpenStack meets this requirement, but not in a very satisfactory way; individual developers may not have access to a ppc64le OpenStack cluster.

from coreos-assembler.

This is fixed right?

from coreos-assembler.

This is fixed right?

@jcajka might be able to say

from coreos-assembler.

@jcajka Can you confirm?

from coreos-assembler.

@cgwalters @sinnykumari I would say as running, init, fetch and build it should be working. I'm currently working on beyond that. Run PR #408 and started to look at oscontainer build atm.
I would defer declaring this fixed on you and @manojnkumar

from coreos-assembler.

Should be long since fixed.

from coreos-assembler.