centos / sig-atomic-buildscripts Goto Github PK

Shell 93.90% Python 4.39% Makefile 1.71%

sig-atomic-buildscripts's Introduction

sig-atomic-buildscripts

This contains metadata and build scripts for the CentOS Atomic Host Development stream. See:

https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel

If you're interested in scripts for the CentOS Core SIG rebuild of EL7, see the "downstream" branch.

Discuss on http://lists.centos.org/pipermail/centos-devel/ and https://lists.projectatomic.io/projectatomic-archives/atomic-devel/

Performing ostree/rpm-ostree updates to CBS

First, ensure the RPM is built in Fedora, rawhide at least, and normally all stable releases. Now:

cd ~/src/distgit/fedora/ostree
rpmbuild-cwd --define 'dist .el7.centos' -bs *.spec
koji -p cbs build atomic7-el7.centos ostree-2016.7-1.el7.centos.src.rpm

sig-atomic-buildscripts's People

Contributors

Stargazers

Watchers

Forkers

red54 baude imcleod cgwalters arilivigni purpleidea keithchambers maxamillion kbsingh tmrts fabiand hellojvm abn dit4c jasonbrooks victorock alex-dr vanloswang sstelfox herlo runcom digideskio gbraad chuanchang puppetninja marszhe bcrochet andrewrothstein miabbott jlebon fillet54 giuseppe andieandpal huzhengchuan johnbieren badis42 liucan01 mikalv sotolitolabs fengyunpan langhorst luofei163 ajsmith rfranke alexxnica kryndex smarterclayton pi-victor pepe-net stonefly128 aekinkjet30 qchojr paulmey doytsujin pinkdiamond1

sig-atomic-buildscripts's Issues

build_stage2.sh failing "undefined symbol: soup_server_listen_all"

I was trying to create atomic bits as suggested in readme but it I am getting a strange error. I tried to debug it but no success.

# bash -x ./build_stage2.sh /srv
++ date +%Y%m%d
+ VERSION=7.20160911
++ date +%Y%m%d_%H%M%S
+ DateStamp=20160911_055713
++ pwd
+ GitDir=/root/work/github/sig-atomic-buildscripts
+ BuildDir=/srv
+ LogFile=/srv/log
+ mkdir -p /srv
++ cd /srv
++ pwd
+ BuildDir=/srv
+ OstreeRepoDir=//srv/repo
+ mkdir -p //srv/repo
+ ln -s //srv/repo /srv/repo
ln: failed to create symbolic link ‘/srv/repo/repo’: File exists
+ set -x
+ set -e
+ set -o pipefail
+ cd /srv
+ systemctl start docker
+ systemctl start libvirtd
+ echo '---------- installer '
+ rpm-ostree-toolbox installer --overwrite --ostreerepo /srv/repo -c /root/work/github/sig-atomic-buildscripts/config.ini -o /srv/installer
+ tee /srv/log
workdir=/tmp/atomic-treecomposefjmljN.tmp
rpmostree_cache_dir=None
pkgdatadir=/usr/share/rpm-ostree-toolbox
os_name=centos-atomic-host
ostree_remote=centos-atomic-host
os_pretty_name=MyCustom CentOS Atomic Host
ostree_repo=/srv/repo
tree_name=standard
tree_file=/home/work/github/sig-atomic-buildscripts/centos-atomic-host.json
arch=x86_64
release=7
ref=centos-atomic-host/7/x86_64/standard
yum_baseurl=http://mirror.centos.org/centos/7/os/x86_64/
lorax_additional_repos=http://mirror.centos.org/centos/7/updates/x86_64/, http://buildlogs.centos.org/centos/7/atomic/x86_64/Packages/, http://cbs.centos.org/repos/atomic7-testing/x86_64/os/
is_final=True
lorax_inherit_repos=None
lorax_exclude_packages=oscap-anaconda-addon
lorax_include_packages=None
lorax_rootfs_size=None
local_overrides=None
http_proxy=None
selinux=True
configdir=/home/work/github/sig-atomic-buildscripts
docker_os_name=mirror.centos.org/centos7-atomic-builder
vsphere_product_name=None
vsphere_product_vendor_name=None
vsphere_product_version=None
vsphere_virtual_system_type=None
ostree: symbol lookup error: ostree: undefined symbol: soup_server_listen_all
Traceback (most recent call last):
  File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose-main", line 75, in <module>
    main()
  File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose-main", line 59, in main
    installer.main(cmd)
  File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/installer.py", line 256, in main
    composer.create(post=args.post)
  File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/taskbase.py", line 495, in create
    self.impl_create(**kwargs)
  File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/installer.py", line 162, in impl_create
    httpd_port = str(trivhttp.http_port)
AttributeError: 'TrivialHTTP' object has no attribute 'http_port'

# rpm -qa | grep ostree
rpm-ostree-toolbox-2016.4-1.el7.centos.x86_64
rpm-ostree-2016.5-1.atomic.el7.x86_64
ostree-2016.7-2.atomic.el7.x86_64
ostree-fuse-2016.7-2.atomic.el7.x86_64

# cat /etc/redhat-release 
CentOS Linux release 7.1.1503 (Core)

Issue with vir7 repo

[virt7-testing]
name=virt7-testing
baseurl=http://cbs.centos.org/repos/virt7-testing/x86_64/os/
gpgcheck=0
exclude=kernel docker

Should be http://cbs.centos.org/repos/virt7-common-testing/x86_64/os/

virt7-testing no longer exists

Missing newer continuous images

Seems like something happened to the every-6-hour job that creates images from the latest continuous tree. E.g. the latest at https://ci.centos.org/artifacts/sig-atomic/centos-continuous/images/cloud/latest/images/ dates from Nov 15th.

redo alpha release process

Right now we're doing:

    ostree --repo=ostree/repo prune --keep-younger-than='30 days ago' --refs-only

But this conflicts with how we're doing alpha promotions. I think we need to either:

Do "recompose" commits like commit --tree=ref= which will create a linear history
Enhance pruning to operate only on the continuous branch
Create separate refs for each alpha tag

I lean a bit more towards the first.

CAHC snapshots

I think most users should be pointed at the downstream build.

We could think of CAHC as like an "alpha". It's for a very small set of developers/testers/contributors for the platform itself.

I think it would make sense to have a "beta" type stream too. When major new features land and we want more users to try it, but not the whole world to update.

My suggested plan here is quite simple: Just create a new ostree ref, "centos-atomic-host/7/x86_64/devel/beta" that's a binary promotion. We can do it manually for now.
We can create images (qcow2 etc.) as well, but I'd suggest we not look at the whole AMI upload and things like that.

What gets tricky is that our CI infrastructure will need some reworking to handle concurrent access to the repo. Right now we have some crude rsync jobs, but we'll need to increase sophistication there.

downstream: enhance release process

@cgwalters We've talked a bit about tagging downstream releases so that we can associate a particular state of the metadata in the repo with a given release. I was looking at how you do alpha releases, but that seems tied up in ci -- do you have thoughts about how we should do this for downstream releases?

Also, @kbsingh and I were talking in the meeting last week about setting up a process where, upon some release tagging event that happens in the repo, we'd have the nightly signed tree that KB is producing cut over to the official release location, allowing us to bring more automation to downstream releases.

Does that sound like something that'd work with the sort of release tagging I mentioned above?

Create images from the smoketested branch

Once https://bugs.centos.org/view.php?id=12686 is fixed, it would be great if we could start creating images from the smoketested branch. TBD whether we should rewire atomic-cahc-image-cloud-continuous to use the smoketested branch instead, or make it an entirely new deliverable.

ci artifacts cleanup and buildlogs

The atomic sig artifacts directory (https://ci.centos.org/artifacts/sig-atomic/) for the centos ci could use a cleanup, most of what's in here is either no longer needed, or will be regenerated when tests run. We are, however, distributing some items from this server that'd be better distributed from a location like buildlogs.centos.org.

@cgwalters would you weigh in on the sort of sync-with-buildlogs / automated cleaning regime that'd work for the items you're responsible for, like the continuous and fedora workstation stuff?

For my part, the downstream folder is what I typically consult, to try out test builds of that release. That particular folder needn't live longer than two weeks at a time.

Rolling in critical fixes for downstream

I think https://rhn.redhat.com/errata/RHSA-2016-2110.html is critical enough that we should do an update, even if upstream hasn't necessarily done one.

CAHC treecompose is failing on 'do-release-tags'

https://ci.centos.org/job/atomic-treecompose-centos7/7155/console

18:03:22 + /home/builder/ostree-releng-scripts/do-release-tags --autocreate --repo=ostree/repo --releases=/home/builder/sig-atomic-buildscripts/releases.yml
18:03:22 Previously: centos-atomic-host/7/x86_64/devel/alpha = d2eee59134a6e11cf364a9dc30f34b27700436bc3a25f81e73ca71eaad67a122, promoted from 2178f1638a702478cc7b555fa25d7b38a0f77a81805f0169945d6f38e809a7e4
18:03:22 Traceback (most recent call last):
18:03:22   File "/home/builder/ostree-releng-scripts/do-release-tags", line 78, in <module>
18:03:22     _,newcommitdata,_ = r.load_commit(newcommit)
18:03:22 GLib.Error: g-io-error-quark: No such metadata object 6097af9f10cbab877d8c9e3689271dfd58c317f1d6084535bf1fc2ba9e9ef24f.commit (1)

get latest oci-systemd-hook and oci-register-machine pkgs into CAHC

$ rpm-ostree status
State: idle
Deployments:
● centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
       Version: 7.2017.3 (2017-01-05 21:17:29)
        Commit: 48917927e1c3ef574dd0a8e285c98775f6227c965da61501f3848d5feba18024
        OSName: centos-atomic-host

  centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
       Version: 7.2017.1 (2017-01-05 14:35:39)
        Commit: 6b9d48311b8e953f07735871d85fc6c74c69c55bd3d93d37d0655a923311bb9a
        OSName: centos-atomic-host

$ rpm -q oci-systemd-hook oci-register-machine
oci-systemd-hook-0.1.4-6.git337078c.el7.x86_64
oci-register-machine-0-1.10.gitfcdbff0.el7.x86_64

Latest commits for these two packages are c6776e8 and dd0daef.

Because of some other issue ( #218 ), these two packages cannot be automatically integrated from github master branch with overlay.

wrong version of skopeo in CAHC

In the latest CAHC build (Version: 7.2016.220 (2016-09-22 14:00:42)), the version of skopeo isn't what I expected to see after #155 was merged.

# rpm -q skopeo
skopeo-0.1.13-8.el7.x86_64

The build artifacts shows that the "right" version was built, but I don't know why it wasn't included in the compose.

https://ci.centos.org/artifacts/sig-atomic/rdgo/centos-continuous/build/

Maybe atomic is dragging in a different version?

Fixate uid/gids

See
https://bugzilla.redhat.com/show_bug.cgi?id=1203006
https://git.fedorahosted.org/cgit/fedora-atomic.git/commit/?id=debbecbc6cec7ae494f26703d0ae28b2c53b0e26

This is not an emergency for CentOS as the latest rpm-ostree will automatically do this given a previous commit - but it means freshly composed trees may be subject to drift.

Talking to docker when i type `atomic host`

Due to something in the chain of imports, I noticed we're connecting to docker.socket when I type atomic host. I don't think we should.

static deltas for release stream

Deltas have been stable upstream for a while - we should make use of them. The CAHC stream already is (although it's not quite as useful there since we're only generating N-1>N).

It's likely just a matter of:

diff --git a/build_stage1.sh b/build_stage1.sh
index 29637fa..c081912 100755
--- a/build_stage1.sh
+++ b/build_stage1.sh
@@ -69,6 +69,7 @@ fi
 ## compose a new tree, based on defs in centos-atomic-host.json

 rpm-ostree compose --repo=${OstreeRepoDir} tree --add-metadata-string=version=${VERSION} ${GitDir}/${TreeFile} |& tee ${BuildDir}/log.compose
+ostree --repo=${OstreeRepoDir} static-delta generate centos-atomic-host/7/x86_64/standard
 ostree --repo=${OstreeRepoDir} summary -u |& tee ${BuildDir}/log.compose

 # deal with https://bugzilla.gnome.org/show_bug.cgi?id=748959

Need a clean way in my script to figure out that its running on atomic

I have a bash script that needs to do something along the lines of

if <running on atomic>
then
 <setup these entries>
else
 #running on standard rhel/centos
  <setup alternate entries>
fi

And this script needs to run in 2 places

During provisioning a Centos/Rhel atomic os as a part of the KS post script
Standalone on the machine after it has booted . Because the user may choose to run it on an existing instance.

Given i need a way in centos to figure out that its running on atomic machine.
I cannot do [ -e "/run/ostree-booted"] as suggested in forums for Case 1 because during provisioning this directory does not exist on an atomic machine.
Any ideas?

For a rhel machine I can do something along the lines of

grep -i "Atomic Host" /etc/os-release 
if [ $? -eq 0 ]
then
 <setup atomic entries>
else
 #running on standard rhel/centos
  <setup alternate entries>
fi

But for centos I cannot get that information because the /etc/os-release looks like


[centos@cloud ~]$ cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

References:
https://bugzilla.redhat.com/show_bug.cgi?id=1200122

Create a new release package

There's a reason that the "Linux distribution" concept ended up being structured the way it has been - it's around "who owns updates", specifically for security.

With CentOS, that's a fairly clear process today - updates flow from Red Hat, the ball passes to CentOS in a fairly transparent way.

With things built in CBS, the issue becomes a lot murkier. While I do think Feodra's trademark guidelines around this are not ideal, they are capturing something important that we're just glossing over - users need to know that CentOS Atomic contains CBS content.

I can take ownership of creating a release package, but let's figure out what it should be called.

old alpha commits pruned

Haven't debugged this but:

[root@localhost ~]# atomic host status
State: idle
Deployments:
● centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/alpha
       Version: 7.2016.447 (2016-10-14 19:14:04)
        Commit: 1cfda3b9d0cccd9559c43e87e114df08a8fd0026821b01617d4327692c322528
        OSName: centos-atomic-host
[root@localhost ~]# atomic host deploy 7.2016.583
Resolving version '7.2016.583'
error: No such metadata object 7ffb9fd5e8d237409e80c8f36f261c20506b17819b7b4e340a3385ae0b98486c.commit

Package installed but files are not available to desired location.

I made below changes to centos-atomic-host.json and atomic-7.1-vagrant.ks also I add another repo ADB to make adb-utils and fuse-sshfs available to repo tree. After running custom build I found out package is installed but files are not present in desired location. Any idea about this behavior?

# git diff atomic-7.1-vagrant.ks
diff --git a/atomic-7.1-vagrant.ks b/atomic-7.1-vagrant.ks
index 8232de2..d65bcfa 100644
--- a/atomic-7.1-vagrant.ks
+++ b/atomic-7.1-vagrant.ks
@@ -126,5 +126,11 @@ EOKEYS
 chmod 600 ~vagrant/.ssh/authorized_keys
 chown -R vagrant:vagrant ~vagrant/.ssh/

+# update docker.service file to exec the certificate generation script
+sed -i.back 's/ExecStart=/ExecStartPre=\/opt\/adb\/cert-gen.sh\n&/' /usr/lib/systemd/system/docker.service
+
+# update the docker config to listen on TCP as well as unix socket
+sed -i.back '/OPTIONS=*/c\OPTIONS="--selinux-enabled -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server-cert.pem --tlskey=/etc/docker/server-key.pem
+
 %end

# git diff centos-atomic-host.json
diff --git a/centos-atomic-host.json b/centos-atomic-host.json
index 81c6b0d..ec500e7 100644
--- a/centos-atomic-host.json
+++ b/centos-atomic-host.json
@@ -5,7 +5,7 @@
     "ref": "centos-atomic-host/7/x86_64/standard",

     "repos": ["CentOS-Base", "CentOS-updates", "CentOS-extras",
-              "rhel-atomic-rebuild", "CentOS-CR"],
+              "rhel-atomic-rebuild", "CentOS-CR", "ADB"],

     "selinux": true,

@@ -60,19 +60,13 @@
@@ -60,19 +60,13 @@
                 "NetworkManager", "vim-minimal", "nano",
                 "sudo",
                 "tuned", "tuned-profiles-atomic",
-                "kubernetes", "etcd",
-                "flannel",
                 "irqbalance",
                 "bash-completion",
-                "rsync", "tmux",
+                "rsync",
                 "net-tools", "nmap-ncat",
-                "bind-utils",
+                "bind-utils", "adb-utils",
                 "authconfig",
                 "policycoreutils-python",
-                "cockpit-shell",
-                "cockpit-bridge",
-                 "cockpit-docker",
-                 "cockpit-ostree",
                 "setools-console",
                 "atomic-devmode",
                 "docker-latest",
@@ -80,7 +74,7 @@
                 "docker-novolume-plugin",
                  "stub-redhat-lsb-core-only-for-ceph",
                  "ceph-common", "device-mapper-multipath",
-                 "sg3_utils", "glusterfs-fuse"],
+                 "sg3_utils", "fuse-sshfs"],

     "remove-from-packages": [["yum", "/usr/bin/.*"],
                             ["kernel", "/lib/modules/.*/drivers/gpu"],

# cat ADB.repo 
[ADB]
name=rhel-atomic-rebuild
baseurl=http://mirror.centos.org/centos/7/atomic/x86_64/adb/
gpgcheck=0

bash-4.2# rpm -aq | grep adb-utils
adb-utils-2.0-1.el7.noarch

bash-4.2# rpm -ql adb-utils
/etc/bash_completion.d/oadm
/etc/bash_completion.d/oc
/etc/bash_completion.d/openshift
/etc/sysconfig/openshift_option
/opt/adb
/opt/adb/add_insecure_registry
/opt/adb/cert-gen.sh
/opt/adb/openshift
/opt/adb/openshift/get_ip_address
/opt/adb/openshift/openshift
/opt/adb/openshift/openshift_provision
/opt/adb/openshift/openshift_stop
/opt/adb/openshift/templates
/opt/adb/openshift/templates/adb
/opt/adb/openshift/templates/adb/image-streams-centos7.json
/opt/adb/openshift/templates/adb/jenkins-template.json
/opt/adb/openshift/templates/cdk
/opt/adb/openshift/templates/cdk/image-streams-rhel7.json
/opt/adb/openshift/templates/cdk/jenkins-ephemeral-template.json
/opt/adb/openshift/templates/cdk/jenkins-persistent-template.json
/opt/adb/openshift/templates/common
/opt/adb/openshift/templates/common/cakephp-mysql.json
/opt/adb/openshift/templates/common/cakephp.json
/opt/adb/openshift/templates/common/eap64-basic-s2i.json
/opt/adb/openshift/templates/common/eap64-mysql-persistent-s2i.json
/opt/adb/openshift/templates/common/jboss-image-streams.json
/opt/adb/openshift/templates/common/jenkins-slave-template.json
/opt/adb/openshift/templates/common/jws30-tomcat7-mysql-persistent-s2i.json
/opt/adb/openshift/templates/common/nodejs-mongodb.json
/opt/adb/openshift/templates/common/nodejs.json
/opt/adb/sccli.py
/opt/adb/sccli.pyc
/opt/adb/sccli.pyo
/usr/bin/add_insecure_registry
/usr/bin/sccli
/usr/lib/systemd/system/openshift.service
/usr/share/doc/adb-utils-2.0
/usr/share/doc/adb-utils-2.0/LICENSE
/usr/share/doc/adb-utils-2.0/README.rst


bash-4.2# ls -lhR /opt/
/opt/:
total 0

CAHC: need newer version of docker due to SELinux problems

Recent composes of CAHC have this error in the logs:

libsemanage.semanage_direct_install_info: Overriding docker module at lower priority 100 with module at priority 400.
15:13:49 Re-declaration of boolean virt_sandbox_use_fusefs
15:13:49 Failed to create node
15:13:49 Bad boolean declaration at /etc/selinux/targeted/tmp/modules/100/virt/cil:159
15:13:49 /usr/sbin/semodule:  Failed!

This seems to arise from the version of selinux-policy-targeted that is being pulled in from the CentOS-CR repo.

@cgwalters suggested on IRC to use a newer version of docker to mitigate this.

Not sure which version to use...maybe something from projectatomic/docker?

redirect alpha → smoketested

Per consensus in SIG Atomic meeting, drop the Alpha branch and redirect it to /smoketested.

Work to make alpha more accessible

The /continuous stream has a very small target audience - basically people who want to test git master. But /alpha has potentially a much larger one, and I think we should get to the point where we say "Yes, it's totally sane to mix some alpha machines into your staging/production".

Tasks for that are the same baseline things for all release streams:

Mirroring
Signing
Security updates (we should have a process to auto-respin alphas when underlying CentOS changes)

unable to run containers with docker-latest in CAHC

# atomic host status
State: idle
Deployments:
● centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
       Version: 7.2017.54 (2017-01-20 16:05:52)
        Commit: efd2a5880903a3fa8b771bd07d05139bbbdccc99e66d06a0a79930da08e34b8b
        OSName: centos-atomic-host
# rpm -q docker-latest
docker-latest-1.13-27.git6cd0bbe.el7.x86_64
# docker run -it centos /bin/bash
/usr/bin/docker-latest: Error response from daemon: linux init cgroups path: mkdir /system.slice: permission denied.

Jan 20 17:18:24 cahc-dev dockerd-latest[12291]: time="2017-01-20T17:18:24.727929701Z" level=info msg="{Action=_ping, Username=vagrant, LoginUID=1001, PID=13295}"
Jan 20 17:18:24 cahc-dev dockerd-latest[12291]: time="2017-01-20T17:18:24.730633933Z" level=info msg="{Action=create, Username=vagrant, LoginUID=1001, PID=13295}"
Jan 20 17:18:24 cahc-dev dockerd-latest[12291]: time="2017-01-20T17:18:24.790314487Z" level=info msg="{Action=attach, ID=97f3af5ea91670143328eac858df6e665a6789c47c5a01349ac64c4437f5cf38, Username=vagrant, LoginUID=1001, PID=13295}"
Jan 20 17:18:24 cahc-dev dockerd-latest[12291]: time="2017-01-20T17:18:24.791365289Z" level=info msg="{Action=start, ID=97f3af5ea91670143328eac858df6e665a6789c47c5a01349ac64c4437f5cf38, Username=vagrant, LoginUID=1001, PID=13295, Config={Hostname=97f3af5ea916, AttachStdin=true, AttachStdout=true, AttachStderr=true, Tty=true, OpenStdin=true, StdinOnce=true, Env=[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin], Cmd=[/bin/bash], ArgsEscaped=false, Image=centos, NetworkDisabled=false, Labels=map[vendor:CentOS build-date:20161214 license:GPLv2 name:CentOS Base Image]}, HostConfig={LogConfig={Type:journald Config:map[]}, NetworkMode=default, PortBindings=map[], RestartPolicy={Name:no MaximumRetryCount:0}, AutoRemove=false, DNS=[], DNSOptions=[], DNSSearch=[], Privileged=false, PublishAllPorts=false, ReadonlyRootfs=false, ShmSize=67108864, Runtime=oci, Resources={CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DiskQuota:0 KernelMemory:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:0xc4210c6270 OomKillDisable:0xc4210c627a PidsLimit:0 Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0}}}"
Jan 20 17:18:25 cahc-dev dockerd-latest[12291]: time="2017-01-20T17:18:25.022894582Z" level=error msg="Handler for POST /v1.25/containers/97f3af5ea91670143328eac858df6e665a6789c47c5a01349ac64c4437f5cf38/start returned error: linux init cgroups path: mkdir /system.slice: permission denied"

problems with ssh key login on CAHC 7.2017.206 after booting cloud image

Initially we observed issues logging into our VMs in OpenStack after booting the centos/7/atomic/continuous image, which we believe to correspond to 7.2017.206.

After the VM was booted, we were being asked to provide a password for the user supplied in the cloud-init data, which was not expected since an SSH key was provided.

An inspection of the logs on the OpenStack VM showed the following denials:

[[32m  OK  [0m] Started Hostname Service.
[    5.517988] type=1107 audit(1490300818.870:6): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc:  denied  { reload } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/iscsi.service" cmdline="/bin/systemctl --no-block reload iscsi.service" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:iscsi_unit_file_t:s0 tclass=service
[    5.517988]  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
[[32m  OK  [0m] Started OpenSSH Server Key Generation.
         Starting OpenSSH server daemon...
[    5.603069] type=1400 audit(1490300818.955:7): avc:  denied  { execute } for  pid=908 comm="11-dhclient" name="chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:chronyd_exec_t:s0 tclass=file
[    5.667488] type=1400 audit(1490300819.020:8): avc:  denied  { getattr } for  pid=908 comm="11-dhclient" path="/usr/libexec/chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:chronyd_exec_t:s0 tclass=file
[[32m  OK  [0m] Started OpenSSH server daemon.
[[32m  OK  [0m] Started LVM2 PV scan on device 252:2.
[    5.698494] type=1400 audit(1490300819.051:9): avc:  denied  { getattr } for  pid=908 comm="11-dhclient" path="/usr/libexec/chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:chronyd_exec_t:s0 tclass=file
[[32m  OK  [0m] Started Dynamic System Tuning Daemon.
[    7.173624] cloud-init[670]: Cloud-init v. 0.7.5 running 'init-local' at Thu, 23 Mar 2017 20:27:00 +0000. Up 7.09 seconds.
[[32m  OK  [0m] Started Initial cloud-init job (pre-networking).
         Starting Initial cloud-init job (metadata service crawler)...
[    8.205079] cloud-init[3036]: Cloud-init v. 0.7.5 running 'init' at Thu, 23 Mar 2017 20:27:01 +0000. Up 8.16 seconds.
[    8.242553] cloud-init[3036]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[    8.243566] cloud-init[3036]: ci-info: +--------+------+----------------+---------------+-------------------+
[    8.244517] cloud-init[3036]: ci-info: | Device |  Up  |    Address     |      Mask     |     Hw-Address    |
[    8.245470] cloud-init[3036]: ci-info: +--------+------+----------------+---------------+-------------------+
[    8.246416] cloud-init[3036]: ci-info: |  lo:   | True |   127.0.0.1    |   255.0.0.0   |         .         |
[    8.247355] cloud-init[3036]: ci-info: | eth0:  | True | 172.16.171.245 | 255.255.255.0 | fa:16:3e:24:b1:f2 |
[    8.248305] cloud-init[3036]: ci-info: +--------+------+----------------+---------------+-------------------+
[    8.249251] cloud-init[3036]: ci-info: ++++++++++++++++++++++++++++++++Route info+++++++++++++++++++++++++++++++++
[    8.250218] cloud-init[3036]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
[    8.251184] cloud-init[3036]: ci-info: | Route | Destination  |   Gateway    |    Genmask    | Interface | Flags |
[    8.252147] cloud-init[3036]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
[    8.264139] cloud-init[3036]: ci-info: |   0   |   0.0.0.0    | 172.16.171.1 |    0.0.0.0    |    eth0   |   UG  |
[    8.265143] cloud-init[3036]: ci-info: |   1   | 172.16.171.0 |   0.0.0.0    | 255.255.255.0 |    eth0   |   U   |
[    8.266141] cloud-init[3036]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+

CentOSDev Atomic Host 7.2017.206 (Devel)
Kernel 3.10.0-514.10.2.el7.x86_64 on an x86_64

host-172-16-171-245 login: [   11.569989] type=1400 audit(1490300824.922:10): avc:  denied  { write } for  pid=5177 comm="passwd" name=".pwd.lock" dev="dm-0" ino=6294770 scontext=system_u:system_r:passwd_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file
[   11.571958] type=1400 audit(1490300824.924:11): avc:  denied  { write } for  pid=5177 comm="passwd" name=".pwd.lock" dev="dm-0" ino=6294770 scontext=system_u:system_r:passwd_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file
[   11.338576] cloud-init[3036]: 2017-03-23 20:27:04,928 - util.py[WARNING]: Failed to disable password for user centos
[   11.339477] cloud-init[3036]: 2017-03-23 20:27:04,929 - util.py[WARNING]: Running users-groups (<module 'cloudinit.config.cc_users_groups' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_users_groups.py'>) failed
[   12.316580] device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.
[   12.940242] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[   12.943291] Bridge firewalling registered
[   12.963912] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[   13.206836] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[   13.261855] type=1107 audit(1490300826.614:12): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc:  denied  { reload } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/iscsi.service" cmdline="/bin/systemctl --no-block reload iscsi.service" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:iscsi_unit_file_t:s0 tclass=service
[   13.261855]  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
[   13.273296] type=1400 audit(1490300826.625:13): avc:  denied  { execute } for  pid=6131 comm="11-dhclient" name="chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:chronyd_exec_t:s0 tclass=file
[   13.276770] type=1400 audit(1490300826.629:14): avc:  denied  { getattr } for  pid=6131 comm="11-dhclient" path="/usr/libexec/chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:chronyd_exec_t:s0 tclass=file
[   13.280280] type=1400 audit(1490300826.633:15): avc:  denied  { getattr } for  pid=6131 comm="11-dhclient" path="/usr/libexec/chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:chronyd_exec_t:s0 tclass=file

I grabbed the same qcow2 from https://ci.centos.org/artifacts/sig-atomic/centos-continuous/images/cloud/ and booted it locally with similar (but not exact) trouble:

[  OK  ] Started Login Service.
[  OK  ] Started GSSAPI Proxy Daemon.
[  OK  ] Reached target NFS client services.
[    4.257100] type=1400 audit(1490302444.420:4): avc:  denied  { append } for  pid=683 comm="chronyd" name="chrony.keys" dev="dm-0" ino=6294783 scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:object_r
:etc_t:s0 tclass=file
[  OK  ] Started Dump dmesg to /var/log/dmesg.
[FAILED] Failed to start NTP client/server.
See 'systemctl status chronyd.service' for details.
[  OK  ] Started Network Manager.
[  OK  ] Reached target Network.
         Starting Logout off all iSCSI sessions on shutdown...
         Starting Notify NFS peers of a restart...
         Starting Enable periodic update of entitlement certificates....
         Starting OpenSSH server daemon...
         Starting System Logging Service...
[  OK  ] Started Containerd Standalone OCI Container Daemon.
         Starting Containerd Standalone OCI Container Daemon...
         Starting Dynamic System Tuning Daemon...
[  OK  ] Started Logout off all iSCSI sessions on shutdown.
[  OK  ] Started Notify NFS peers of a restart.
[  OK  ] Started Enable periodic update of entitlement certificates..
[  OK  ] Started System Logging Service.
[  OK  ] Started Authorization Manager.
         Starting Network Manager Script Dispatcher Service...
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Remote File Systems.
         Starting Crash recovery kernel arming...
         Starting Permit User Sessions...
         Starting Availability of block devices...
[  OK  ] Started OpenSSH server daemon.
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Availability of block devices.
[  OK  ] Started Network Manager Script Dispatcher Service.
[  OK  ] Started Command Scheduler.
         Starting Command Scheduler...
[  OK  ] Started Serial Getty on ttyS0.
         Starting Serial Getty on ttyS0...
[  OK  ] Started Getty on tty1.
         Starting Getty on tty1...
[  OK  ] Reached target Login Prompts.
         Starting Hostname Service...
[  OK  ] Started Hostname Service.
[    4.659630] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    5.041939] type=1107 audit(1490302445.204:5): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc:  denied  { reload } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/
iscsi.service" cmdline="/bin/systemctl --no-block reload iscsi.service" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:iscsi_unit_file_t:s0 tclass=service
[    5.041939]  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
[    5.100316] type=1400 audit(1490302445.263:6): avc:  denied  { execute } for  pid=890 comm="11-dhclient" name="chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=sys
tem_u:object_r:chronyd_exec_t:s0 tclass=file
[    5.117110] type=1400 audit(1490302445.280:7): avc:  denied  { getattr } for  pid=890 comm="11-dhclient" path="/usr/libexec/chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:s0
 tcontext=system_u:object_r:chronyd_exec_t:s0 tclass=file
[    5.133489] type=1400 audit(1490302445.296:8): avc:  denied  { getattr } for  pid=890 comm="11-dhclient" path="/usr/libexec/chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:s0
 tcontext=system_u:object_r:chronyd_exec_t:s0 tclass=file
[  OK  ] Started Dynamic System Tuning Daemon.
[    5.507361] device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.
[  OK  ] Started Device-mapper event daemon.
         Starting Device-mapper event daemon...
[  OK  ] Started LVM2 PV scan on device 252:2.
[    5.921956] cloud-init[672]: Cloud-init v. 0.7.5 running 'init-local' at Thu, 23 Mar 2017 20:54:06 +0000. Up 5.87 seconds.
[  OK  ] Started Initial cloud-init job (pre-networking).
         Starting Initial cloud-init job (metadata service crawler)...
[    6.989367] cloud-init[1934]: Cloud-init v. 0.7.5 running 'init' at Thu, 23 Mar 2017 20:54:07 +0000. Up 6.94 seconds.
[    7.025875] cloud-init[1934]: ci-info: ++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[    7.030828] cloud-init[1934]: ci-info: +--------+------+-----------------+---------------+-------------------+
[    7.033286] cloud-init[1934]: ci-info: | Device |  Up  |     Address     |      Mask     |     Hw-Address    |
[    7.035669] cloud-init[1934]: ci-info: +--------+------+-----------------+---------------+-------------------+
[    7.037411] cloud-init[1934]: ci-info: |  lo:   | True |    127.0.0.1    |   255.0.0.0   |         .         |
[    7.039371] cloud-init[1934]: ci-info: | eth0:  | True | 192.168.122.245 | 255.255.255.0 | 52:54:00:74:a8:56 |
[    7.048494] cloud-init[1934]: ci-info: +--------+------+-----------------+---------------+-------------------+
[    7.050983] cloud-init[1934]: ci-info: +++++++++++++++++++++++++++++++++Route info++++++++++++++++++++++++++++++++++
[    7.052240] cloud-init[1934]: ci-info: +-------+---------------+---------------+---------------+-----------+-------+
[    7.053819] cloud-init[1934]: ci-info: | Route |  Destination  |    Gateway    |    Genmask    | Interface | Flags |
[    7.055280] cloud-init[1934]: ci-info: +-------+---------------+---------------+---------------+-----------+-------+
[    7.056720] cloud-init[1934]: ci-info: |   0   |    0.0.0.0    | 192.168.122.1 |    0.0.0.0    |    eth0   |   UG  |
[    7.058219] cloud-init[1934]: ci-info: |   1   | 192.168.122.0 |    0.0.0.0    | 255.255.255.0 |    eth0   |   U   |
[    7.067453] cloud-init[1934]: ci-info: +-------+---------------+---------------+---------------+-----------+-------+
[  OK  ] Started Initial cloud-init job (metadata service crawler).
         Starting Docker Storage Setup...
[  OK  ] Reached target Cloud-config availability.
         Starting Apply the settings specified in cloud-config...
[    7.846152] cloud-init[1948]: Cloud-init v. 0.7.5 running 'modules:config' at Thu, 23 Mar 2017 20:54:07 +0000. Up 7.76 seconds.
         Stopping OpenSSH server daemon...
[  OK  ] Stopped OpenSSH server daemon.
[  OK  ] Started Docker Storage Setup.
         Starting Docker Application Container Engine...
         Starting OpenSSH server daemon...
[  OK  ] Started OpenSSH server daemon.
[  OK  ] Started Apply the settings specified in cloud-config.
         Starting Execute cloud user/final scripts...
[    8.306802] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[    8.355703] Bridge firewalling registered
[    8.386575] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    8.722403] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[    8.839013] type=1107 audit(1490302449.001:9): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc:  denied  { reload } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/
iscsi.service" cmdline="/bin/systemctl --no-block reload iscsi.service" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:iscsi_unit_file_t:s0 tclass=service
[    8.839013]  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
[    8.914273] type=1400 audit(1490302449.077:10): avc:  denied  { execute } for  pid=2830 comm="11-dhclient" name="chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=s
ystem_u:object_r:chronyd_exec_t:s0 tclass=file
[    8.920517] type=1400 audit(1490302449.084:11): avc:  denied  { getattr } for  pid=2830 comm="11-dhclient" path="/usr/libexec/chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:
s0 tcontext=system_u:object_r:chronyd_exec_t:s0 tclass=file
[    8.930289] type=1400 audit(1490302449.093:12): avc:  denied  { getattr } for  pid=2830 comm="11-dhclient" path="/usr/libexec/chrony-helper" dev="dm-0" ino=4257533 scontext=system_u:system_r:NetworkManager_t:
s0 tcontext=system_u:object_r:chronyd_exec_t:s0 tclass=file
[  OK  ] Started Docker Application Container Engine.
[    9.171731] cloud-init[2086]: Cloud-init v. 0.7.5 running 'modules:final' at Thu, 23 Mar 2017 20:54:09 +0000. Up 9.07 seconds.
ci-info: no authorized ssh keys fingerprints found for user centos.
[    9.217236] cloud-init[2086]: ci-info: no authorized ssh keys fingerprints found for user centos.
[    9.222757] cloud-init[2086]: 2017-03-23 20:54:09,382 - util.py[WARNING]: Running ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python2.7/site-packages/cloudi
nit/config/cc_ssh_authkey_fingerprints.py'>) failed
ec2: 
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 256 e1:eb:fd:1b:29:da:4e:3f:2d:32:ac:f1:0b:e0:30:79   (ECDSA)
ec2: 256 47:b2:a9:d7:0c:3f:06:73:74:09:c7:ca:7d:5b:2b:ab   (ED25519)
ec2: 2048 08:72:a8:d8:3c:59:20:4a:06:6d:f6:94:2e:84:aa:98   (RSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE6n70U6x7t33R+626QS2Q7fDj1Pq389y5+dqnbAs7WIx/pXY1jLOeNuyn/xJTXeHHJvA/dhKKS5xZp17hU/n3o= 
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG5kpXt91i1BYw9MAK+KqIe9XkYGTuU11EhSQrRiJHnL 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0RgF0yEUhzSkjpvbQMn/aWT+cs8qgw4MYMCPJ6eqVHUeGgrhD1mtbLOl5u2HkR10vG7e8cU3hRYQnlKnvksAShII4TahnxISZSu5Nd/8lEKOTotebzzJ8iFRp+JTUFWjjNDzlChV5yjxPrO4OF72GbxuKDkI9lEXUPgFmgu9G3+4
KNYxpTiPZ7QFaIfVHcyUDo++++01YcuBcAbxT0C3MQV8aLGH/3maoVQX1apqDaCUph0fABFEo0Q/sGU5ThETYHgRaD3C63EW+d7pIU7fXucPDYOw+dienpKFBdrJNgYNDgWofl76O/Sw9+6iDNLkRF5hZyF3891myQrBuPEPh 
-----END SSH HOST KEY KEYS-----
[    9.260107] cloud-init[2086]: Cloud-init v. 0.7.5 finished at Thu, 23 Mar 2017 20:54:09 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0][dsmode=local].  Up 9.25 seconds
[FAILED] Failed to start Execute cloud user/final scripts.
See 'systemctl status cloud-final.service' for details.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.

RE: CentOS 7 x86_64 Devel AtomicHost EBS HVM 20150306_01 - ami-a522b0d2

Quick question. I'm using AWS eu-west-1 image. Below is run on fresh install.

Do I manually have to copy kubelet.service to /etc/systemd/system as specified here? http://www.projectatomic.io/blog/2014/11/testing-kubernetes-with-an-atomic-host/
Can't find a way to enable etcd. Above instruction assumes it should be there. Mar 16 11:47:02 ip-10-100-55-173.eu-west-1.compute.internal systemd[25392]: Failed at step CHDIR spawning /usr/bin/etcd: No such file or directory >>> i'm attaching full stacktrace below.

[centos@ip-10-100-55-173 ~]$ sudo -s
bash-4.2# cp /usr/lib/systemd/system/kubelet.service /etc/systemd/system/
bash-4.2# systemctl daemon-reload
bash-4.2# for SERVICES in etcd kube-apiserver kube-controller-manager  kube-scheduler docker kube-proxy.service  kubelet.service; do 
>         systemctl restart $SERVICES
>         systemctl enable $SERVICES
>         systemctl status $SERVICES
>     done
ln -s '/usr/lib/systemd/system/etcd.service' '/etc/systemd/system/multi-user.target.wants/etcd.service'
etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled)
   Active: failed (Result: exit-code) since Mon 2015-03-16 11:47:02 UTC; 135ms ago
 Main PID: 25392 (code=exited, status=200/CHDIR)

Mar 16 11:47:02 ip-10-100-55-173.eu-west-1.compute.internal systemd[1]: Started Etcd Server.
Mar 16 11:47:02 ip-10-100-55-173.eu-west-1.compute.internal systemd[25392]: Failed at step CHDIR spawning /usr/bin/etcd: No such file or directory
Mar 16 11:47:02 ip-10-100-55-173.eu-west-1.compute.internal systemd[1]: etcd.service: main process exited, code=exited, status=200/CHDIR
Mar 16 11:47:02 ip-10-100-55-173.eu-west-1.compute.internal systemd[1]: Unit etcd.service entered failed state.
ln -s '/usr/lib/systemd/system/kube-apiserver.service' '/etc/systemd/system/multi-user.target.wants/kube-apiserver.service'
kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled)
   Active: active (running) since Mon 2015-03-16 11:47:02 UTC; 187ms ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 25405 (kube-apiserver)
   CGroup: /system.slice/kube-apiserver.service
           └─25405 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd_servers=http://127.0.0.1:4001 --address=127.0.0.1 --port=8080 --kubelet_port=10250 --allow_privileged=false --portal_net=10.254.0.0/16

Mar 16 11:47:02 ip-10-100-55-173.eu-west-1.compute.internal systemd[1]: Started Kubernetes API Server.
ln -s '/usr/lib/systemd/system/kube-controller-manager.service' '/etc/systemd/system/multi-user.target.wants/kube-controller-manager.service'
kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled)
   Active: active (running) since Mon 2015-03-16 11:47:02 UTC; 314ms ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 25421 (kube-controller)
   CGroup: /system.slice/kube-controller-manager.service
           └─25421 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --machines=127.0.0.1 --master=127.0.0.1:8080

Mar 16 11:47:02 ip-10-100-55-173.eu-west-1.compute.internal systemd[1]: Started Kubernetes Controller Manager.
Mar 16 11:47:03 ip-10-100-55-173.eu-west-1.compute.internal kube-controller-manager[25421]: I0316 11:47:03.147782   25421 plugins.go:70] No cloud provider specified.
ln -s '/usr/lib/systemd/system/kube-scheduler.service' '/etc/systemd/system/multi-user.target.wants/kube-scheduler.service'
kube-scheduler.service - Kubernetes Scheduler Plugin
   Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled)
   Active: active (running) since Mon 2015-03-16 11:47:03 UTC; 270ms ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 25437 (kube-scheduler)
   CGroup: /system.slice/kube-scheduler.service
           └─25437 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=127.0.0.1:8080

Mar 16 11:47:03 ip-10-100-55-173.eu-west-1.compute.internal systemd[1]: Started Kubernetes Scheduler Plugin.
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: active (running) since Mon 2015-03-16 11:47:04 UTC; 145ms ago
     Docs: http://docs.docker.com
 Main PID: 25464 (docker)
   CGroup: /system.slice/docker.service
           └─25464 /usr/bin/docker -d --selinux-enabled --storage-opt dm.fs=xfs --storage-opt dm.datadev=/dev/mapper/atomicos-docker--data --storage-opt dm.metadatadev=/dev/mapper/atomicos-docker--meta

Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal docker[25464]: time="2015-03-16T11:47:03Z" level="info" msg="+job serveapi(unix:///var/run/docker.sock)"
Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal docker[25464]: time="2015-03-16T11:47:03Z" level="info" msg="Listening for HTTP on unix (/var/run/docker.sock)"
Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal docker[25464]: time="2015-03-16T11:47:04Z" level="info" msg="+job init_networkdriver()"
Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal docker[25464]: time="2015-03-16T11:47:04Z" level="info" msg="-job init_networkdriver() = OK (0)"
Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal docker[25464]: time="2015-03-16T11:47:04Z" level="info" msg="Loading containers: start."
Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal docker[25464]: time="2015-03-16T11:47:04Z" level="info" msg="Loading containers: done."
Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal docker[25464]: time="2015-03-16T11:47:04Z" level="info" msg="docker daemon: 1.5.0 a8a31ef/1.5.0; execdriver: native-0.2; graphdriver: devicemapper"
Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal docker[25464]: time="2015-03-16T11:47:04Z" level="info" msg="+job acceptconnections()"
Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal docker[25464]: time="2015-03-16T11:47:04Z" level="info" msg="-job acceptconnections() = OK (0)"
Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal systemd[1]: Started Docker Application Container Engine.
ln -s '/usr/lib/systemd/system/kube-proxy.service' '/etc/systemd/system/multi-user.target.wants/kube-proxy.service'
kube-proxy.service - Kubernetes Kube-Proxy Server
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled)
   Active: active (running) since Mon 2015-03-16 11:47:04 UTC; 238ms ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 25516 (kube-proxy)
   CGroup: /system.slice/kube-proxy.service
           ├─25516 /usr/bin/kube-proxy --logtostderr=true --v=0 --etcd_servers=http://127.0.0.1:4001
           └─25545 /usr/bin/kube-proxy --logtostderr=true --v=0 --etcd_servers=http://127.0.0.1:4001

Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal systemd[1]: Started Kubernetes Kube-Proxy Server.
Mar 16 11:47:04 ip-10-100-55-173.eu-west-1.compute.internal kube-proxy[25516]: I0316 11:47:04.825636   25516 proxier.go:328] Initializing iptables
ln -s '/etc/systemd/system/kubelet.service' '/etc/systemd/system/multi-user.target.wants/kubelet.service'
kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled)
   Active: active (running) since Mon 2015-03-16 11:47:05 UTC; 217ms ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 25549 (kubelet)
   CGroup: /system.slice/kubelet.service
           └─25549 /usr/bin/kubelet --logtostderr=true --v=0 --etcd_servers=http://127.0.0.1:4001 --address=127.0.0.1 --port=10250 --hostname_override=127.0.0.1 --allow_privileged=false

Mar 16 11:47:05 ip-10-100-55-173.eu-west-1.compute.internal systemd[1]: Started Kubernetes Kubelet Server.
Mar 16 11:47:05 ip-10-100-55-173.eu-west-1.compute.internal kubelet[25549]: W0316 11:47:05.200516   25549 server.go:240] Could not load kubernetes auth path: stat : no such file or directory. Continuing with defaults.
Mar 16 11:47:05 ip-10-100-55-173.eu-west-1.compute.internal kubelet[25549]: I0316 11:47:05.200972   25549 node.go:52] Connecting to docker on unix:///var/run/docker.sock
Mar 16 11:47:05 ip-10-100-55-173.eu-west-1.compute.internal kubelet[25549]: I0316 11:47:05.201637   25549 server.go:305] No api server defined - no events will be sent.
Mar 16 11:47:05 ip-10-100-55-173.eu-west-1.compute.internal kubelet[25549]: I0316 11:47:05.201697   25549 server.go:356] Watching for etcd configs at [http://127.0.0.1:4001]
Mar 16 11:47:05 ip-10-100-55-173.eu-west-1.compute.internal kubelet[25549]: I0316 11:47:05.202600   25549 logs.go:40] etcd DEBUG: get /registry/nodes/127.0.0.1/boundpods [http://127.0.0.1:4001]
Mar 16 11:47:05 ip-10-100-55-173.eu-west-1.compute.internal kubelet[25549]: I0316 11:47:05.202632   25549 logs.go:40] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/nodes/127.0.0.1/boundpods?consist...sorted=false
Mar 16 11:47:05 ip-10-100-55-173.eu-west-1.compute.internal kubelet[25549]: I0316 11:47:05.202642   25549 logs.go:40] etcd DEBUG: send.request.to http://127.0.0.1:4001/v2/keys/registry/nodes/127.0.0.1/boundpo...| method GET
Mar 16 11:47:05 ip-10-100-55-173.eu-west-1.compute.internal kubelet[25549]: I0316 11:47:05.215398   25549 event.go:117] Event(api.ObjectReference{Kind:"Minion", Namespace:"default", Name:"127.0.0.1", UID:"127...ing kubelet.
Hint: Some lines were ellipsized, use -l to show in full.

CAHC: docker returning 'Unknown runtime specified oci' starting existing containers

Our automated tests failed when running atomic run on an existing container in CAHC 7.2017.268:

https://s3.amazonaws.com/aos-ci/atomic-host-tests/improved-sanity-test/cahc/latest/improved-sanity-test.log

The output was not enough to pin down the reason, so I reproduced manually and saw the following:

# docker start centos_httpd
Error response from daemon: Unknown runtime specified oci
Error: failed to start containers: centos_httpd

The test built the centos_httpd container using the version (1.12.6-14.gitf499e8b.el7) of docker in CAHC 7.2017.267 and then after the upgrade, we see this error. The newer version of docker is 1.12.6-16.el7.

The treecompose for this versions seems to indicate that docker is now coming from CentOS-extras

https://ci.centos.org/job/atomic-treecompose-centos7/6819/console

Previously, it was coming from virt7-docker-common-candidate

https://ci.centos.org/job/atomic-treecompose-centos7/6818/console

Test images before publishing

In the same vein as #228, we should establish some sort of sanity test before updating the public image.

The great thing about this is that from there, we could then run those same sanity tests in PR testers on various projects, knowing that we have a good baseline. E.g. running the improved sanity test would have caught projectatomic/atomic#838 at PR time.

Ensure we get atomic from git

Right now it's not due to versioning issues. https://trello.com/c/aMWxazPK/363-5-ci-libhif-priorities-fix-atomic-command-version

Atomic vagrant box doesn't work with private_network

Vagrantfile created by vagrant init centos/atomic-host and private network enabled:

Vagrant.configure("2") do |config|
  config.vm.box = "centos/atomic-host"
  config.vm.network "private_network", ip: "192.168.33.10"
end

Result:

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'centos/atomic-host'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'centos/atomic-host' is up to date...
==> default: Setting the name of the VM: centos-atomic_default_1490027763982_65539
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: hostonly
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: No guest additions were detected on the base box for this VM! Guest
    default: additions are required for forwarded ports, shared folders, host only
    default: networking, and more. If SSH fails on this machine, please install
    default: the guest additions and repackage the box to continue.
    default: 
    default: This is not an error message; everything may continue to work properly,
    default: in which case you may ignore this message.
==> default: Configuring and enabling network interfaces...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

# Down the interface before munging the config file. This might
# fail if the interface is not actually set up yet so ignore
# errors.
/sbin/ifdown 'enp0s8'
# Move new config into place
mv -f '/tmp/vagrant-network-entry-enp0s8-1490027783-0' '/etc/sysconfig/network-scripts/ifcfg-enp0s8'
# attempt to force network manager to reload configurations
nmcli c reload || true

# Restart network
service network restart


Stdout from the command:

Restarting network (via systemctl):  [FAILED]


Stderr from the command:

usage: ifdown <configuration>
Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details.

Network interfaces:

$ vagrant ssh -c "ifconfig -a"
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:34:82:4d:79  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::2390:d464:7556:7596  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:bd:c2:7a  txqueuelen 1000  (Ethernet)
        RX packets 420  bytes 53125 (51.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 297  bytes 50859 (49.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.33.10  netmask 255.255.255.0  broadcast 192.168.33.255
        inet6 fe80::a00:27ff:feab:9c44  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:ab:9c:44  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22  bytes 2196 (2.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 74  bytes 6324 (6.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 74  bytes 6324 (6.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Error messages:

Mar 20 16:37:59 localhost.localdomain systemd[1]: Starting LSB: Bring up/down networking...
-- Subject: Unit network.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit network.service has begun starting up.
Mar 20 16:37:59 localhost.localdomain network[12956]: Bringing up loopback interface:  [  OK  ]
Mar 20 16:37:59 localhost.localdomain network[12956]: Bringing up interface enp0s8:  [  OK  ]
Mar 20 16:38:00 localhost.localdomain NetworkManager[712]: <info>  [1490027880.1176] audit: op="connection-activate" uuid="5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03" name="System eth0" result="fail" reason="No suitable device found for thi
Mar 20 16:38:00 localhost.localdomain network[12956]: Bringing up interface eth0:  Error: Connection activation failed: No suitable device found for this connection.
Mar 20 16:38:00 localhost.localdomain network[12956]: [FAILED]
Mar 20 16:38:00 localhost.localdomain network[12956]: RTNETLINK answers: File exists
Mar 20 16:38:00 localhost.localdomain network[12956]: RTNETLINK answers: File exists
Mar 20 16:38:00 localhost.localdomain network[12956]: RTNETLINK answers: File exists
Mar 20 16:38:00 localhost.localdomain network[12956]: RTNETLINK answers: File exists
Mar 20 16:38:00 localhost.localdomain network[12956]: RTNETLINK answers: File exists
Mar 20 16:38:00 localhost.localdomain network[12956]: RTNETLINK answers: File exists
Mar 20 16:38:00 localhost.localdomain network[12956]: RTNETLINK answers: File exists
Mar 20 16:38:00 localhost.localdomain network[12956]: RTNETLINK answers: File exists
Mar 20 16:38:00 localhost.localdomain network[12956]: RTNETLINK answers: File exists
Mar 20 16:38:00 localhost.localdomain systemd[1]: network.service: control process exited, code=exited status=1
Mar 20 16:38:00 localhost.localdomain systemd[1]: Failed to start LSB: Bring up/down networking.

Vagrant Box

$ vagrant box update
==> default: Checking for updates to 'centos/atomic-host'
    default: Latest installed version: 7.20170131
    default: Version constraints: 
    default: Provider: virtualbox
==> default: Box 'centos/atomic-host' (v7.20170131) is running the latest version.

downstream: Include GPG signature in delta from 7.20170209

See ostreedev/ostree#748

Develop Atomic distribution for ARM

Given that there's a lot of interest from our users for CentOS Atomic Host on ARM (for IoT uses), let's figure out what would be involved in developing a regular release for ARM.

What build hardware do we need that we don't already have?
Do we care about ARMv7, or just ARMv8/ARM64?
This will also require container builds for ARM; what's required to make that happen that we don't already have?
What else am I not thinking of?

NetworkManager unable to create /etc/resolv.conf; SELinux denial

We hadn't seen this when booting an older version of the CAHC cloud image (7.2016.834) and then deploying the newer commits. However, once the cloud-image job was fixed (see #194), the internal CAHC tests started to fail at the provisioning stage.

The initial error was was related to pulling the commit metadata:

failed: [10.8.169.180] => {"changed": true, "cmd": ["ostree", "pull", "--commit-metadata-only", "--depth=1", "centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous"], "delta": "0:00:00.280108", "end": "2016-12-09 13:18:30.029410", "rc": 1, "start": "2016-12-09 13:18:29.749302", "warnings": []}
08:18:46 stderr: error: Error resolving 'ci.centos.org': Name or service not known

However, further investigation points to an empty /etc/resolv.conf. The following messages are seen in the journal:

Dec 08 20:46:00 rhel-atomic-7 kernel: type=1400 audit(1481229960.565:5): avc:  denied  { create } for  pid=717 comm="NetworkManager" name="resolv.conf.IBFURY" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file
Dec 08 20:46:00 rhel-atomic-7 NetworkManager[717]: <warn>  [1481229960.5715] dns-mgr: could not commit DNS changes: Failed to create file '/etc/resolv.conf.IBFURY': Permission denied

This was originally reported in the internal #atomic channel by @p3ck

CAHC: consider adding a parallel debug stream

E.g. centos-atomic-host/7/x86_64/devel/continuous-debug. This would include the debuginfo RPMs of the gitoverlayed pkgs. Bonus points would be if the packages were actually compiled with CFLAGS="-g -Og" and gdb were added to the manifest. Since the continuous stream is primarily meant for users and developers to test upcoming features, we're bound to require frequent debugging as we get bug reports and verify functionality.

One might say we could just layer the debuginfo pkgs from the atomic-centos-continuous repo, but those only contain the RPMs for the last build, which might not always be what you want if you explicitly want to work in a previous version (e.g. when debugging something like coreos/rpm-ostree#306).

/cc @cgwalters

docker error - Unknown lvalue 'TasksMax' in section 'Service'

Saw this today when testing latest CAHC. I spoke with @lsm5 on #atomic and he said he was working on a fix.

$ sudo journalctl -u docker -b
-- Logs begin at Tue 2017-01-03 17:01:19 UTC, end at Wed 2017-01-11 19:49:22 UTC. --
Jan 11 19:47:17 cahc-dev systemd[1]: [/usr/lib/systemd/system/docker.service:16] Unknown lvalue 'TasksMax' in section 'Service'

$ rpm -q docker
docker-1.12.5-15.git079fbe3.el7.x86_64

$ atomic host status
State: idle
Deployments:
● centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
       Version: 7.2017.17 (2017-01-11 19:08:58)
        Commit: a28e2f7aa6a5f29cf6f23361afd26f18dc2ace4fd707f63f8af2f48aacde0596
        OSName: centos-atomic-host

  centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
       Version: 7.2017.11 (2017-01-09 18:15:03)
        Commit: 68b594696d7677a82ae6deed6fad1b4f5f7f38535b725dbeab2841f38178841c
        OSName: centos-atomic-host

Having the Amazon ECS Agent packaged up would improve container experience for users in AWS

https://github.com/aws/amazon-ecs-agent needs to be packaged up in a way that we can roll out an AWS ECS specific atomichost image; this would allow users on AWS ECS serivice to better normalise their CentOS Atomic Host experience with the in platform integrations provided in AWS.

Need summary files for centos repo

I dont see a summary file at http://mirror.centos.org/centos/7/atomic/x86_64/repo/ .. Would be ideal to have that.

Other notes

walters | jbrooks, can we add `ostree --repo=repo summary -u` in the builds? 
jbrooks | walters, Sure, can you point me to how/where fedora does it?                            
jbrooks | assuming they do                                                                        
walters | that is literally it, just after a commit is updated, run that command                  
walters | the reason it's not automatic is to allow multiple commits to be updated atomically     
walters | and it needs to be serialized                                                           
jbrooks | And it automatically becomes part of the repo?                                          
walters | it's a bit like `createrepo`                                                            
walters | right                                                                                   
jbrooks | Yeah, I can add that

'No upgrade available' using recent CAHC composes

Trying to use atomic host upgrade on a CAHC Vagrant box is returning the No upgrade available message. But I can pull the commit data and then atomic host deploy to the newest version.

# atomic host status
State: idle
Deployments:
● centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
       Version: 7.2017.73 (2017-02-07 17:57:45)
        Commit: 3dfa9dbcec7419dd283ba07be3e8b784c8f516d6c5b72bb0981baff5b6f45c0d
        OSName: centos-atomic-host

  centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
       Version: 7.2017.39 (2017-01-18 04:12:17)
        Commit: 4c3ebe3ac906f1e8af681f46d0d738cb14beede8b3479c27f7231569251a9b4e
        OSName: centos-atomic-host
# rpm -q ostree rpm-ostree
ostree-2017.1.16-3d38f03e4fd4cc20f754bb787feb0d109387f4f8.25e721f59f276f93487c85db723e473c1db1ef1e.el7.centos.x86_64
rpm-ostree-2017.1.15-687567d3eef470887ff14f9128ab3be2fa25d68d.f4d2fc43cd0bf3e9b4455d88afb9f04983396013.el7.centos.x86_64
# atomic host upgrade
No upgrade available.
# ostree pull --commit-metadata-only --depth=1 centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous

6 metadata, 0 content objects fetched; 460 B transferred in 1 seconds                                                                                                                                              
# ostree log centos-atomic-host/7/x86_64/devel/continuous
commit 76475f2f350d5bedfeca5555597d57313c376d5247c795fda229f4f9974a455c
Date:  2017-02-07 21:10:19 +0000
Version: 7.2017.75
(no subject)

commit 43caa89e1d7f59f7f3a553d914b8432a75b6293bd30ddf6f91ec5a1772f889be
Date:  2017-02-07 18:29:12 +0000
Version: 7.2017.74
(no subject)

commit 3dfa9dbcec7419dd283ba07be3e8b784c8f516d6c5b72bb0981baff5b6f45c0d
Date:  2017-02-07 17:57:45 +0000
Version: 7.2017.73
(no subject)

<< History beyond this commit not fetched >>
# atomic host deploy 7.2017.75
Resolving version '7.2017.75'

22 metadata, 26 content objects fetched; 88633 KiB transferred in 16 seconds                                                                                                                                       
Copying /etc changes: 27 modified, 4 removed, 48 added
Transaction complete; bootconfig swap: yes deployment count change: 0
Freed objects: 580.7 MB
Changed:
  bubblewrap 0.1.7.8-a27841ed094b7db7a1cada2086c4bfc4d7ddd842.el7.centos -> 0.1.7.9-a2ceebb38c1ff7ab90d2229c83aa00d14a5a8fce.el7.centos
  ostree 2017.1.16-3d38f03e4fd4cc20f754bb787feb0d109387f4f8.25e721f59f276f93487c85db723e473c1db1ef1e.el7.centos -> 2017.1.19-425ccc0a33610c71f0258423c83701dc4e273ee7.9912ee7f1a162f3ce8a81d4645b6a7f0ff0913f0.el7.
centos
  ostree-grub2 2017.1.16-3d38f03e4fd4cc20f754bb787feb0d109387f4f8.25e721f59f276f93487c85db723e473c1db1ef1e.el7.centos -> 2017.1.19-425ccc0a33610c71f0258423c83701dc4e273ee7.9912ee7f1a162f3ce8a81d4645b6a7f0ff0913f
0.el7.centos
Added:
  ostree-libs-2017.1.19-425ccc0a33610c71f0258423c83701dc4e273ee7.9912ee7f1a162f3ce8a81d4645b6a7f0ff0913f0.el7.centos.x86_64
Run "systemctl reboot" to start a reboot

/usr/bin/docker-{current,latest} not properly labeled

CAHC is currently hitting https://bugzilla.redhat.com/show_bug.cgi?id=1358819. I.e. docker-current and docker-latest are not properly labeled. This is resulting in a failure to start any container:

[root@c7-aht ~]# docker run --rm busybox echo hello world
permission denied
docker: Error response from daemon: Container command could not be invoked..
[root@c7-aht ~]# journalctl -k | tail | grep avc
Jan 09 17:46:42 c7-aht kernel: type=1400 audit(1483984002.857:444): avc:  denied  { transition } for  pid=12425 comm="exe" path="/bin/echo" dev="dm-4" ino=4195329 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c404,c898 tclass=process
[root@c7-aht ~]# ls -laZ /usr/bin/docker*
-rwxr-xr-x. root root system_u:object_r:docker_exec_t:s0 /usr/bin/docker
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-current
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/dockerd-latest
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-latest
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-latest-storage-setup
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-proxy
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-storage-setup
-rwx------. root root system_u:object_r:bin_t:s0       /usr/bin/docker-v1.10-migrator-helper
-rwx------. root root system_u:object_r:bin_t:s0       /usr/bin/docker-v1.10-migrator-local
[root@c7-aht ~]# rpm -q container-selinux
container-selinux-2.2-1.el7.noarch
[root@c7-aht ~]# matchpathcon /usr/bin/docker-current
/usr/bin/docker-current system_u:object_r:bin_t:s0

Though container-selinux-2.2-1.el7.noarch does indeed provide the correct contexts. Most likely something is going wrong in its %post at compose time.

Image creation tasks currently failing

Seems like imagefactory times out on disk activity

https://ci.centos.org/job/atomic-cahc-image-cloud-continuous/

Related, would be nice if published the screenshot somewhere on error.

vagrant: sshfs hack failed networking at build

grep curl /var/log/anaconda/program.log 
21:33:07,890 INFO program: 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: kojipkgs.fedoraproject.org; Unknown error

I wonder if this is something to do with the fact that centos and fedora are colocated? I hit issues related to this before. Hm, no, works fine when I ssh to a duffy node. 😕

CAHC: ancient version of atomic getting pulled in

The internal CAHC tests started failing (yesterday, I think) and when I finally got around to looking at the problem, I found that the version of atomic is ancient:

# rpm -q atomic
atomic-0-0.10.gite5734c4.el7.x86_64

Looking at the treecompose job, it appears that the compose started getting the RPM from the atomic7-testing repo:

14:12:38   atomic-0-0.10.gite5734c4.el7.x86_64 (atomic7-testing)

Current CAHC's selinux-policy is from 7.2 but everything else is 7.3

centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
       Version: 7.2016.1060 (2016-12-02 15:22:23)
        Commit: 6f9faa916eca6fe332fe8d4e2273186ab5d6fb08600c85db3c3d15b9cff47aa4
        OSName: centos-atomic-host

libselinux-utils-2.5-6.el7.x86_64
libselinux-2.5-6.el7.x86_64
selinux-policy-3.13.1-63.atomic.el7.7.noarch
libselinux-python-2.5-6.el7.x86_64
docker-selinux-1.10.3-46.el7.14.x86_64
selinux-policy-targeted-3.13.1-63.atomic.el7.7.noarch
NetworkManager-1.4.0-12.el7.x86_64
NetworkManager-libnm-1.4.0-12.el7.x86_64

Seeing a lot of these selinux errors.

Error: type=1400 audit(1480703043.439:4): avc:  denied  { create } for  pid=713 comm="NetworkManager" name="resolv.conf.X7B2RY" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file

docker panic in cloud image builds

https://ci.centos.org/view/Atomic/job/atomic-cahc-image-cloud-continuous/841/console

13:25:44 Running: docker run --rm --privileged --workdir /out --net=none -v /home/builder/sig-atomic-buildscripts:/in:ro -v /tmp/atomic-treecomposeXpXybp.tmp/tmp-kickstart:/out rpm-ostree-toolbox/centos-atomic-host-7-kickstart-app
13:25:44 panic: standard_init_linux.go:175: exec user process caused "permission denied" [recovered]
13:25:44 	panic: standard_init_linux.go:175: exec user process caused "permission denied"
13:25:44 
13:25:44 goroutine 1 [running, locked to thread]:
13:25:44 panic(0x6f2ea0, 0xc42016c230)
13:25:44 	/usr/lib/golang/src/runtime/panic.go:500 +0x1a1
13:25:44 github.com/urfave/cli.HandleAction.func1(0xc420097748)
13:25:44 	/builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x247
13:25:44 panic(0x6f2ea0, 0xc42016c230)
13:25:44 	/usr/lib/golang/src/runtime/panic.go:458 +0x243
13:25:44 github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc420097198, 0xc4200260c8, 0xc420097238)
13:25:44 	/builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x18f
13:25:44 github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc420057630, 0xaac9c0, 0xc42016c230)
13:25:44 	/builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353
13:25:44 main.glob..func8(0xc42009aa00, 0x0, 0x0)
13:25:44 	/builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/main_unix.go:26 +0x66
13:25:44 reflect.Value.call(0x6ddc20, 0x769b48, 0x13, 0x73c049, 0x4, 0xc420097708, 0x1, 0x1, 0x4d1728, 0x731ea0, ...)
13:25:44 	/usr/lib/golang/src/reflect/value.go:434 +0x5c8
13:25:44 reflect.Value.Call(0x6ddc20, 0x769b48, 0x13, 0xc420097708, 0x1, 0x1, 0xac2700, 0xc4200976e8, 0x4da706)
13:25:44 	/usr/lib/golang/src/reflect/value.go:302 +0xa4
13:25:44 github.com/urfave/cli.HandleAction(0x6ddc20, 0x769b48, 0xc42009aa00, 0x0, 0x0)
13:25:44 	/builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x1e0
13:25:44 github.com/urfave/cli.Command.Run(0x73c215, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74d852, 0x51, 0x0, ...)
13:25:44 	/builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xc3b
13:25:44 github.com/urfave/cli.(*App).Run(0xc4200be000, 0xc42000c120, 0x2, 0x2, 0x0, 0x0)
13:25:44 	/builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x611
13:25:44 main.main()
13:25:44 	/builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/main.go:137 +0xbd6
13:25:45 Traceback (most recent call last):
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose-main", line 75, in <module>
13:25:45     main()
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose-main", line 57, in main
13:25:45     imagefactory.main(cmd)
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/imagefactory.py", line 574, in main
13:25:45     imageouttypes=imagetypes
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/taskbase.py", line 495, in create
13:25:45     self.impl_create(**kwargs)
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/imagefactory.py", line 386, in impl_create
13:25:45     ksdata = self.formatKS(ksfile)
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/imagefactory.py", line 321, in formatKS
13:25:45     run_sync(cmd, env=child_env)
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/utils.py", line 36, in run_sync
13:25:45     subprocess.check_call(args, **kwargs)
13:25:45   File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
13:25:45     raise CalledProcessError(retcode, cmd)
13:25:45 subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '--privileged', '--workdir', '/out', '--net=none', '-v', '/home/builder/sig-atomic-buildscripts:/in:ro', '-v', '/tmp/atomic-treecomposeXpXybp.tmp/tmp-kickstart:/out', 'rpm-ostree-toolbox/centos-atomic-host-7-kickstart-app']' returned non-zero exit status 2

rpm-ostree install fails running post

[cloud-user@cloud-6 ~]$ sudo rpm-ostree status
State: idle
Deployments:
● centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
Version: 7.2016.1087 (2016-12-05 17:23:23)
Commit: 5bc514a4b818aaf7b4da43721796577ca790bab6ab7b141e5909ca84bdc5a226
OSName: centos-atomic-host

centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
Version: 7.2016.71 (2016-08-23 13:09:11)
Commit: 4df97c9e62f711f26c5b6f92607fbbd32df89223edb2678a54a87f67ec36d86f
OSName: centos-atomic-host
[cloud-user@cloud-6 ~]$ sudo rpm-ostree install wget
Checking out tree 5bc514a... done

Downloading metadata: [========================================================================================] 100%
Resolving dependencies... done
Overlaying... done
Running %post for wget...... error: Running %post for wget: Executing bwrap: Child process exited with code 1

Atomic: update kickstart file

There are several upstream changes we should take, such as GROWPART. This should fix projectatomic/container-storage-setup#106 (comment)

CAHC autobuilds broken due to fedora cert change

See https://lists.fedoraproject.org/archives/list/[email protected]/thread/KZOF6KSEBZFHD643PAPLYEYVW6OZWPNG/

sync docker images to artifacts

So we don't use space on the master.

centos-release-atomic patch

The current, downstream-7.20170117, release of centos atomic is subject to this issue with rpm-ostree package layering: https://bugzilla.redhat.com/show_bug.cgi?id=1399770

The upcoming downstream release includes rpm-ostree-client-2016.13-1.atomic.el7.x86_64, and doesn't have this issue, but @cgwalters has suggested that we adopt a patch similar to https://bugzilla.redhat.com/show_bug.cgi?id=1399770#c2 for centos-release-atomic.

Colin's patch puts os-release into /usr/lib/ and symlinks that to /etc/. One potential issue here is that centos-release-atomic depends on centos-release, and that's where our os-release comes from. https://git.centos.org/blob/rpms!centos-release-atomic/b132798c53c3ebe5544790053b59af19529307c1/SPECS!centos-release-atomic.spec

@kbsingh Should we adopt a release pkg approach similar to rhel, I don't recall all the reasons for doing this the way we have?

rdgo currently failing on go compiler

https://ci.centos.org/job/atomic-rdgo-centos7/12699/console

19:49:22 ERROR: Command failed: 
19:49:22  # /usr/bin/yum-builddep --installroot /var/lib/mock/centos-and-extras-7-x86_64-mockchain-14870/root/ --releasever 7 /var/lib/mock/centos-and-extras-7-x86_64-mockchain-14870/root//builddir/build/SRPMS/runc-1.0.0-5.rc2.gitc91b5be.el7.centos.src.rpm
19:49:22 Getting requirements for runc-1.0.0-5.rc2.gitc91b5be.el7.centos.src
19:49:22  --> libseccomp-devel-2.3.1-2.el7.x86_64
19:49:22  --> golang-github-cpuguy83-go-md2man-1.0.4-2.el7_2.x86_64
19:49:22 Error: No Package found for compiler(go-compiler)

RFC: Add a new 'smoketested' branch

As described in the ostree manual, it would be interesting to have a new "smoketested" branch based off of the continuous branch gated on tests.

Ideally, these tests would run in the CentOS CI as well. However, because we cannot currently very easily provision Atomic Hosts in the CI infra, I propose that for now, we make use of the automated tests done by the internal Red Hat Jenkins. These jobs run the improved-sanity-test on every new continuous commit.

An easy way to implement this workflow would be to automatically push a new smoketested tag to the releases.yml file whenever such a test passes. Once better support for Atomic Host provisioning is added directly in the CentOS CI, we can look into migrating the testers there.