openshift / ci-operator Goto Github PK

View Code? Open in Web Editor NEW

28.0 16.0 58.0 17.74 MB

Go 99.82% Makefile 0.10% Dockerfile 0.07%

ci-operator's People

Contributors

Stargazers

Watchers

Forkers

smarterclayton csrwng enterstudio spadgett petr-muller derekwaynecarr stevekuznetsov droslean paulfantom rabugopsl sallyom kwoodson jcantrill mjudeikis stuartchuan tsakaloss deads2k twiest charlesakalugwu vrutkovs wewang58 wking flaper87 mfojtik ecordell benjaminapetersen richm pmorie jcaianirh jlebon sanchezl rogbas booxter chancez shiywang dcbw abhinavdahiya bparees soltysh mandre hongkailiu olegvn88 cynepco3hahue p0lyn0mial bardielle global-localhost global19 global19-atlassian-net yuvalk isabella232 emilvberglind srcarrier cjschaef yunjiang29 sarroutbi opokornyy nunnatsa romanbednar

ci-operator's Issues

artifacts: annotation for logs when something else fails

In general, if the e2e tests container fails, we will often want to get the logs from the installer as well, even if the installer container did not fail. We should thread this through as an annotation like ci-operator.openshift.io/wait-for-container-artifacts and slap the setup container into there.

/cc @vrutkovs @smarterclayton

We need reference for config file

People often ask about a reference documentation for config files. #57 had an ambition to add one but it was not straightforward enough and multiple different proposals were raised there.

The actual content is already written in the form of comments of https://github.com/openshift/ci-operator/blob/master/pkg/api/types.go. It could be possible to generate the reference from there using godoc , as advised in #57 by @stevekuznetsov and @smarterclayton (see discussion and the actual generated content). Personally, I would be more in favor of having a real doc rather than sending a Go code documentation to people.

Avoid duplicating common logic in Openshift templates

We need a way to reuse a single Openshift template which provides us a fresh cluster installed on Azure. We want to be able to execute actions before and after the cluster is installed. Examples for running before include building a node image that the install can use. Examples for running logic after include running all sorts of e2e tests or upgrades.

Strawman implementation: ci-operator reads a golang template and based on parameters provided into its env, it generates the actual Openshift template to use. Templates all the way down.

Another alternative was brought up by @stevekuznetsov in one of the latter ci meetings. IIUC, he alluded to adding cluster provisioning logic in the ci-operator so we could avoid using templates.

Thoughts?

@bbguimaraes @droslean @petr-muller @smarterclayton @stevekuznetsov

@openshift/sig-azure fyi

error: failed to create objects: rolebindings.authorization.openshift.io "e2e-gcp-image-puller" already exists

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_openshift-ansible/9533/pull-ci-openshift-openshift-ansible-e2e-gcp/1218/

2018/08/10 19:50:16 Resolved source https://github.com/openshift/openshift-ansible to master@bbb01630, merging: #9533 80f0245f @sdodson
2018/08/10 19:50:16 Resolved openshift/release:golang-1.9 to sha256:5be0666e469c9ecd31f801b384aa4744426532110901cac978ad6754408c83fe
2018/08/10 19:50:16 Resolved openshift/origin-v3.11:base to sha256:84f6e97ce7022cab256e377f968cca6e1dc3e8b9f6d5aa48ffb1d2b08f2e9a04
2018/08/10 19:50:16 Running [input:root], [input:base-without-rpms], [release-inputs], src, rpms, [serve:rpms], base, ansible, [output:stable:ansible], [images], e2e-gcp
2018/08/10 19:50:16 Creating namespace ci-op-wmkcfr5y
2018/08/10 19:50:16 Setting a soft TTL of 1h0m0s for the namespace
2018/08/10 19:50:16 Setting a hard TTL of 12h0m0s for the namespace
2018/08/10 19:50:16 Updated secret e2e-gcp-cluster-profile
2018/08/10 19:50:16 Tagging release images from openshift/origin-v3.11:${component}
2018/08/10 19:50:16 Tagging openshift/release:golang-1.9 into pipeline:root
2018/08/10 19:50:16 Tagging openshift/origin-v3.11:base into pipeline:base-without-rpms
2018/08/10 19:50:16 Building src
2018/08/10 19:50:16 Build src already succeeded in 26s
2018/08/10 19:50:16 Building rpms
2018/08/10 19:50:16 Build rpms already succeeded in 43s
2018/08/10 19:50:16 RPMs being served at http://rpm-repo-ci-op-wmkcfr5y.svc.ci.openshift.org/
2018/08/10 19:50:16 Building base
2018/08/10 19:50:16 Build base already succeeded in 10s
2018/08/10 19:50:16 Building ansible
2018/08/10 19:50:16 Build ansible already succeeded in 4m27s
2018/08/10 19:50:17 Tagging ansible into stable
2018/08/10 19:50:17 All images ready
2018/08/10 19:50:17 Executing template e2e-gcp
2018/08/10 19:50:17 Waiting for template instance e2e-gcp to be deleted ...
2018/08/10 19:51:39 Container artifacts in pod e2e-gcp-crio completed successfully
2018/08/10 19:51:39 Container setup in pod e2e-gcp-crio completed successfully
2018/08/10 19:51:39 Container teardown in pod e2e-gcp-crio completed successfully
2018/08/10 19:51:39 Container test in pod e2e-gcp-crio completed successfully
2018/08/10 21:19:57 Ran for 1h29m41s
error: failed to create objects: rolebindings.authorization.openshift.io "e2e-gcp-image-puller" already exists
pods "e2e-gcp" already exists

cannot get secrets in the temporary namespace

$ oc get secrets
Error from server (Forbidden): secrets is forbidden: User "kargakis" cannot list secrets in the namespace "ci-op-2zfnzjw2": no RBAC policy matched

Trying to debug a template that uses secrets but I don't have access to them.

@stevekuznetsov @droslean

/kind bug

Provide examples for common use cases

I guess this is part of a greater documentation effort but we need examples for most common use cases (eg. a job that builds binaries, a job that builds and runs unit tests, etc.). The only source of examples atm is openshift/release but that shouldn't be the case.

cc @petr-muller

Let the GitHub PR Author have view (or edit?) access into the namespace created by their tests

This helps people iterate on stuff if they login to the api.ci cluster

/cc @bbguimaraes @smarterclayton

Allow external container images as base

It seems I can only use imagestreamtags to base my builds on. It would be nice to allow the use of container images served outside of the cluster. Or maybe this is already supported but I have no idea how to make use of it.

@stevekuznetsov

`clonerefs` output is ignored, so tests can run with with incorrect refs

We ignore the output of clonerefs due to 7ad9489 and 9aa7a8a.

That's simply wrong and must be fixed.

We should either run initupload or write the log and read it ourselves, the output is simple.

If we had done this, the master-dockerless-build-temporary issue would not have happened.

/assign @smarterclayton

steps ... are missing dependencies ...

config.json with base_rpm_images without base_images throws:

2018/06/20 17:37:58 Resolved openshift/openshift-ansible@master to commit c3195db0363833c8a273b1a9bc91c4e6021477ee
2018/06/20 17:37:58 Resolved test-namespace/release-with-clonerefs:latest to sha256:3f6da5c4da6a463f53bed269ff94ac12d7d7ce67cbf4b16141219a69f8bc9c32
2018/06/20 17:37:58 Resolved test-namespace/centos:7 to sha256:eed5b251b615d1e70b10bcec578d64e8aa839d2785c2ffd5424e472818c42755
2018/06/20 17:37:58 Ran for 0s

error: steps ([images], [output:stable:installer], [output:stable:installer], installer, [images], [images], [output:stable:installer], [images], [output:stable:installer], [output:stable:ins
taller], installer, [images], [images], [output:stable:installer], [images], [output:stable:installer], [output:stable:installer], installer, [images], [images], [images], [output:stable:inst
aller], [images]) are missing dependencies (<*api.externalImageLink>, <*api.internalImageLink>, <*api.internalImageLink>, <*api.internalImageLink>, <*api.externalImageLink>, <*api.externalIma
geLink>, <*api.internalImageLink>, <*api.externalImageLink>, <*api.internalImageLink>, <*api.internalImageLink>, <*api.internalImageLink>, <*api.externalImageLink>, <*api.externalImageLink>, 
<*api.internalImageLink>, <*api.externalImageLink>, <*api.internalImageLink>, <*api.internalImageLink>, <*api.internalImageLink>, <*api.externalImageLink>, <*api.externalImageLink>, <*api.ext
ernalImageLink>, <*api.internalImageLink>, <*api.externalImageLink>)

determinize-prow-jobs invalidates TEST_SKIP

We trying to skip some invalid/broken/not-relevant tests for openshift-azure. But we have failure in determinize-prow-jobs:
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_release/1520/pull-ci-openshift-release-master-ordered-prow-config/544

But if we commit with "determinized" content we get invalid regexp:

.*((The HAProxy router should set Forwarded headers appropriately)|(oc            new-app should succeed with a --name of 58 characters)|(process valueFrom            in build strategy environment variables should successfully resolve valueFrom            in docker build environment variables)|(prune builds based on settings            in the buildconfig should prune failed builds based on the failedBuildsHistoryLimit            setting)|(Prometheus when installed to the cluster should start and expose            a secured proxy and unsecured metrics)|(Prometheus when installed to the            cluster should start and expose a secured proxy and verify build metrics)|(templateservicebroker)|(deploymentconfigs            with multiple image change triggers [Conformance] should run a successful            deployment with multiple triggers)|(templateinstance readiness test should            report failed soon after an annotated objects has failed)|(s2i build with            a root user image should create a root build and pass with a privileged            SCC)).*

Which has a lot of whitespaces and skips are ignored.

Target substitution breaks jobs that override `RPM_REPO`

error: could not run steps: could not get Route for RPM server: routes.route.openshift.io "rpm-repo" not found

Job.

Failed to push `release` image

failed to wait for release pod to complete: could not wait for pod completion: the pod ci-op-cx46zrrm/release-latest failed after 27s (failed containers: release):  unknown

Container release exited with code 1, reason Error
---
info: Using registry public hostname registry.svc.ci.openshift.org
info: Manifests will be extracted to /tmp/release-image-2018-09-14T182540Z286090734
warning: Could not load current user information: user: unknown userid 1018740000
Uploading ... 1.092kB/s
error: unable to upload layer sha256:256b176beaff7815db2a93ee2071621ae88f451bb1e198ca73010ed5bba79b65 to destination repository: blob mounted from: ci-op-cx46zrrm/stable@sha256:256b176beaff7815db2a93ee2071621ae88f451bb1e198ca73010ed5bba79b65 to: {application/octet-stream 74694681 sha256:256b176beaff7815db2a93ee2071621ae88f451bb1e198ca73010ed5bba79b65 []}

Happened twice in https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_openshift-ansible/10031/pull-ci-openshift-openshift-ansible-master-gcp-major-upgrade/36 and https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_openshift-ansible/10031/pull-ci-openshift-openshift-ansible-master-gcp-major-upgrade/35/. gcp task has been running at the time this happened

Email notifications

We would like to move some of our periodic jobs from jenkins to CI-Operator

We can't do this because of lack of email notification capability.

@stevekuznetsov is this on the roadmap somewhere? (Seen the upstream issue 8668 already)

Onboarding doc needs to be updated to mention split Prow job config files

ONBOARD.md was written while we still had a giant config file in openshift/release. We now have an ability to split them and we probably want new onboarders to create new configs this way.

https://github.com/openshift/ci-operator/blob/master/ONBOARD.md#add-prow-jobs

Make API patterns look more like kube pattterns

ci-operator might eventually be useful for openshift users. If we have any possibility of using it there, we should be consistent with Kube API conventions, specifically camelCase instead of snake_case and also version the object.

@stevekuznetsov

Feature: Add support for other build strategies?

I spend some time trying to set up a build strategy that wasn't the Docker build strategy, but it looks like we currently hard-code the Docker strategy. Are their plans to expose other strategies?

lint plugin is not working for some users.

Even though that the lint run successfully, the bot coulnd't comment back, failling with an error like:

{"author":"droslean","component":"hook","error":"status code 422 not one of [200], body: {\"message\":\"Validation Failed\",\"errors\":[\"User is blocked\"],\"documentation_url\":\"https://developer.github.com/v3/pulls/reviews/#create-a-pull-request-review\"}","event-GUID":"5911cd40-b5bf-11e8-9ca4-f93c1bc79c7d","event-type":"issue_comment","level":"error","msg":"Error handling GenericCommentEvent.","org":"openshift","plugin":"golint","pr":120,"repo":"ci-operator","time":"2018-09-11T12:36:58Z","url":"https://github.com/openshift/ci-operator/pull/120#issuecomment-420258619"}

Have canonical inputs to config.json that can be overriden

config.json takes a number of input images (that today must come from a cluster)

{
  "tag_specification": {
    "namespace": "openshift",
    "name": "origin-v3.10",
    "tag": "",
    "tag_overrides": {}
  },
  "base_rpm_images": [
    {
      "namespace": "openshift",
      "name": "origin-v3.10",
      "tag": "base",
      "as": "base"
    }
  ],
  "test_base_image": {
    "namespace": "ci",
    "name": "release-with-clonerefs",
    "tag": "golang-1.9"
  },

These are all details of the particular cluster a job is run against, not a detail about the project itself. each of these steps results in something being tagged into either stable or pipeline with a short name (which must be unique).

I think the canonical form of these should either be in the dockerfiles themselves or as regular docker image references (docker.io/openshift/origin-base:v3.10) where the CI system can override them by name (use namespace openshift name origin-v3.10 as 'base') and a local developer running CI operator against an arbitrary cluster can see it simply work.

Suggested change:

change base_rpm_images and test_base_image to be something a bit more generic:

{
  "images": {
    "test-base": {"image": "docker.io/openshift/origin-release:golang-1.9"},
    "base": {"image": "docker.io/openshift/origin-base:latest"} 
  },
  "tag_specification": {
     "cluster": "api.ci.openshift.org",
    "namespace": "openshift",
    "name": "origin-v3.10"
  },
}

which can also be overridden via command line flag --images:

... --images="$( cat images.json )"
$ cat images.json
{
  "images": {
    "test-base": {"namespace": "ci", "name": "release-with-clonerefs", "tag": "golang-1.9"},
    "base": {"namespace": "openshift", "origin-v3.10", "tag": "base"},
  },
  "tag_specification": {
    "namespace": "openshift",
    "name": "origin-v3.10",
    "tag": "",
    "tag_overrides": {}
  },
}

The CI system would selective override images and tag specification. The end user would be able to run ci-operator locally.

Need to make promote resilient to conflict errors

https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/branch-ci-openshift-cluster-samples-operator-images-master/4

Forgot to do that

We should be using YAML for our configuration

Instead of the JSON files that we currently support. The k8s apimachinery YAML parsing library is the most appropriate one for us to use here.

/kind feature

Dependency resolution mode

We need an entrypoint that will take job config and determine which image(s) are necessary, waiting on or attempting to build only those.

Secret injection flow

We would like to run our E2E tests, which requires some secrets to be injected. Env variables, configMaps.

We need a way to create those secrets. Currently, they exist in our azure namespace. Ci Operator could copy them to temporary namespace when referred.
It would be great if we could provide PodSpec struct for volumes/secrets/configMaps so we define how things are exposed to the test/build container.

@stevekuznetsov if you think there is a way to achieve this now let me know :) I might miss it in the api model

Allow tagging images with the git commit they were built from

We want to publish images tagged with the git commit they were built from. Not sure how difficult it is but my first thought was to allow referencing JOB_SPEC inside the ci-operator config. Then we could do something like:

  "images": [
    {
      "from": "base",
      "to": "$JOB_SPEC.refs.base_sha",
      "dockerfile_path": "Dockerfile.sync",
      "inputs": {
        "bin": {"paths": [{"source_path": "/go/src/github.com/openshift/openshift-azure/sync", "destination_dir": "."}]},
        "root": {"as": ["sync"]}
      }
    }
  ],

See openshift/openshift-azure#327 for more info.

@stevekuznetsov @bbguimaraes @droslean @openshift/sig-azure

Annotation to force log output containerwise.

Currently, ci-operator.openshift.io/always-show-output annotation specifies if the pod will always show the logs no matter what the exit code is. We need to have the same feature, but containerwise.

Add new annotation ci-operator.openshift.io/always-show-container-output so specify which container logs ci-operator will output after the pod will fail.

Support other tests types for openshift-ansible

openshift_ansible test type would provision cluster-launch-e2e template. In order to move tasks to ci-operator-prowgen ci-operator should support other test types:

openshift_ansible_upgrade (cluster-launch-e2e-upgrade template)
openshift_ansible_custom (cluster-launch-e2e-openshift-ansible template`)

Rename `test_base_image` to something more intuitive

test_base_image feels like the base image that is going to be used for running tests. Seems to be something different (the base image used for the binary builds).

@stevekuznetsov @smarterclayton

ci-operator tries to reuse terminating namespaces

2018/09/18 09:29:42 Resolved source https://github.com// to @
2018/09/18 09:29:42 Resolving inputs for the test
2018/09/18 09:29:42 Resolved openshift/release:golang-1.9 to sha256:8332bbe6875aaca035568f9808b7f2146d4e02e7f122669a0c442ea6106a9a50
2018/09/18 09:29:42 Resolved openshift/origin-v3.10:base to sha256:ac253e2792e493b24e632d0ac48c607ce1b250d3b410e1665f250c6965d3a57e
2018/09/18 09:29:42 Resolved inputs, targetting namespace ci-op-2zfnzjw2
2018/09/18 09:29:42 Running [release-inputs], build-azure-image
2018/09/18 09:29:42 Creating namespace ci-op-2zfnzjw2
2018/09/18 09:29:47 Ran for 5s
error: could not initialize namespace: cannot retrieve test namespace: projects.project.openshift.io "ci-op-2zfnzjw2" is forbidden: User "system:serviceaccount:ci:ci-operator" cannot get projects.project.openshift.io in the namespace "ci-op-2zfnzjw2": no RBAC policy matched
{"component":"entrypoint","error":"wrapped process failed: exit status 1","level":"error","msg":"Error executing test process: %v.wrapped process failed: exit status 1","time":"2018-09-18T09:29:47Z"}

ci-op-2zfnzjw2 was in terminating phase and the ci-operator tried to reuse it. If a namespace is terminating, ci-operator should probably create a different one or wait until it's cleaned up to recreate it.

Documentation

This repo lacks a README.md and a repo title.

ci-operator should write events when it completes and when it creates a namespace

So that the event stream contains a record.

@stevekuznetsov now that we are doing a better job at recording events.

onboarding doc should provide inventory/description of pertinent ISTs in api.ci

A list of the imagestreamtags in the openshift namespace in api.ci that can be referenced in your ci-operator's config.json, along with information about the contents of each IST, or which openshif/origin Dockerfile they are built from, or how they get updated as PRs in openshift/origin merge, would have proven very useful during my jenkins test-e2e work.

Can we use dep?

Or is it still impossible to vendor OpenShift client libraries with dep?

/cc @smarterclayton @deads2k

Have a more Travis/PAPR-style interface

There's a natural tension between Prow which was designed for Kubernetes, where a centralized repo wants its tests owned by a core set of people, versus a testing system like Travis which is designed to do completely arbitrary things for tens (hundreds?) of thousands of git repositories.

Today AIUI, the Prow/CI configurations are stored in configmaps which are themselves stored in a git repo (right?). Let's say that we wanted to adopt a CI-operator/Prow like flow across a vast array of repositories that make up https://github.com/coreos/ and https://github.com/projectatomic/

We really need an easy and obvious way for repo owners to own their builds, just like Travis and PAPR.

One really cool thing about the CI operator flow is giving a PR an entire OpenShift namespace. I love that idea. If we had a flow where one could do e.g.:

# ci/.osci.yml
container: registry.fedoraproject.org/fedora:28
exec: ci/run.sh

#!/usr/bin/bash
set -xeuo pipefail
yum -y install ...
yum -y install origin-clients
mkdir -p ~/.kube
cp /run/secrets/osci-kubeconfig ~/.kube/config

# Just use this container, or sometimes use oc to create further test pods...

That'd be very cool. A key advantage of this over the current flow is that if I want to test a change to my CI configuration, I can do that as a pull request on my own project - it's how Travis and PAPR work today.

v3.11 images being published to docker hub under the "latest" tag

I have twice now observed a :latest image pulled from dockerhub that contained v3.11 content.

@stevekuznetsov @smarterclayton

/cc @deads2k

Deleting a pod while the template instance is running doesn't abort the job

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_installer/170/pull-ci-origin-installer-e2e-aws-all/1/

ci-operator should have failed, but instead it looks like it hung waiting for something

src build hung for over 2 hours, even after namespace deleted

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/20824/pull-ci-origin-e2e-gcp/3138/

2018/09/02 01:19:40 Resolved source https://github.com/openshift/origin to master@07c07088, merging: #20824 0907d128 @smarterclayton
2018/09/02 01:19:40 Resolving inputs for the test
2018/09/02 01:19:40 Resolved ci/src-cache-origin:master to sha256:4573cac3532b6fed399f1d78eb287202bdf133511756a70a4a912bbcf8743cf2
2018/09/02 01:19:40 Resolved openshift/centos:7 to sha256:fc2476ccae2a5186313f2d1dadb4a969d6d2d4c6b23fa98b6c7b0a1faad67685
2018/09/02 01:19:40 Resolved inputs, targetting namespace ci-op-7yznljfn
2018/09/02 01:19:40 Running [input:root], [input:os], [release-inputs], src, base, [output:stable:base], bin, rpms, [serve:rpms], base-with-rpms, pod, template-service-broker, cli, hypershift, hyperkube, egress-router, egress-http-proxy, egress-dns-proxy, keepalived-ipfailover, control-plane, tests, deployer, recycler, docker-builder, haproxy-router, f5-router, node, [output:stable:pod], [output:stable:template-service-broker], [output:stable:cli], [output:stable:hypershift], [output:stable:hyperkube], [output:stable:egress-router], [output:stable:egress-http-proxy], [output:stable:egress-dns-proxy], [output:stable:keepalived-ipfailover], [output:stable:control-plane], [output:stable:tests], [output:stable:deployer], [output:stable:recycler], [output:stable:docker-builder], [output:stable:haproxy-router], [output:stable:f5-router], [output:stable:node], [images], e2e-gcp
2018/09/02 01:19:40 Creating namespace ci-op-7yznljfn
2018/09/02 01:19:40 Setting a soft TTL of 1h0m0s for the namespace
2018/09/02 01:19:40 Setting a hard TTL of 12h0m0s for the namespace
2018/09/02 01:19:40 Setting up pipeline imagestream for the test
2018/09/02 01:19:40 Populating secrets for test
2018/09/02 01:19:40 Updated secret e2e-gcp-cluster-profile
2018/09/02 01:19:40 Tagging ci/src-cache-origin:master into pipeline:root
2018/09/02 01:19:40 Tagging openshift/centos:7 into pipeline:os
2018/09/02 01:19:40 Tagged release images from openshift/origin-v3.11:${component}, images will be pullable from registry.svc.ci.openshift.org/ci-op-7yznljfn/stable:${component}
2018/09/02 01:19:40 Building src
SIGABRT: abort
PC=0x459121 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x19b4048, 0x0, 0x0, 0x0, 0x7ffc00000000, 0x0, 0x0, 0x0, 0x7ffc47e5d4c8, 0x410a8b, ...)
	/usr/local/go/src/runtime/sys_linux_amd64.s:530 +0x21
runtime.futexsleep(0x19b4048, 0x7ffc00000000, 0xffffffffffffffff)
	/usr/local/go/src/runtime/os_linux.go:45 +0x4b
runtime.notesleep(0x19b4048)
	/usr/local/go/src/runtime/lock_futex.go:151 +0x9b
runtime.stoplockedm()
	/usr/local/go/src/runtime/proc.go:2101 +0x8c
runtime.schedule()
	/usr/local/go/src/runtime/proc.go:2493 +0x2da
runtime.park_m(0xc42046fb00)
	/usr/local/go/src/runtime/proc.go:2604 +0xb6
runtime.mcall(0x0)
	/usr/local/go/src/runtime/asm_amd64.s:351 +0x5b

goroutine 1 [select, 131 minutes]:
github.com/openshift/ci-operator/pkg/steps.Run(0x129fe80, 0xc4203a5c40, 0xc420650840, 0x3, 0x4, 0x118ba00, 0xc, 0xc420332240, 0x8f)
	/go/src/github.com/openshift/ci-operator/pkg/steps/run.go:45 +0x548
main.(*options).Run.func3(0xc400000008, 0x12198d8)
	/go/src/github.com/openshift/ci-operator/cmd/ci-operator/main.go:439 +0x5aa
github.com/openshift/ci-operator/pkg/interrupt.(*Handler).Run(0xc4202b38f0, 0xc42047fea8, 0x0, 0x0)
	/go/src/github.com/openshift/ci-operator/pkg/interrupt/interrupt.go:105 +0xff
main.(*options).Run(0xc420142000, 0x0, 0x0)
	/go/src/github.com/openshift/ci-operator/cmd/ci-operator/main.go:405 +0x3b0
main.main()
	/go/src/github.com/openshift/ci-operator/cmd/ci-operator/main.go:150 +0x144

goroutine 5 [chan receive]:
github.com/openshift/ci-operator/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x19b3240)
	/go/src/github.com/openshift/ci-operator/vendor/github.com/golang/glog/glog.go:879 +0x8b
created by github.com/openshift/ci-operator/vendor/github.com/golang/glog.init.0
	/go/src/github.com/openshift/ci-operator/vendor/github.com/golang/glog/glog.go:410 +0x203

goroutine 66 [syscall, 131 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 67 [select, 131 minutes, locked to thread]:
runtime.gopark(0x121b3c0, 0x0, 0x1187d49, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc420468f50, 0xc4200be780)
	/usr/local/go/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 6 [chan receive, 131 minutes]:
github.com/openshift/ci-operator/pkg/interrupt.(*Handler).Run.func2(0xc42001ade0, 0xc4202b38f0)
	/go/src/github.com/openshift/ci-operator/pkg/interrupt/interrupt.go:98 +0x3d
created by github.com/openshift/ci-operator/pkg/interrupt.(*Handler).Run
	/go/src/github.com/openshift/ci-operator/pkg/interrupt/interrupt.go:97 +0xcf

goroutine 32 [chan receive, 131 minutes]:
github.com/openshift/ci-operator/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x128d920, 0xc420455f20, 0xc420454f90)
	/go/src/github.com/openshift/ci-operator/vendor/k8s.io/client-go/tools/record/event.go:229 +0x98
created by github.com/openshift/ci-operator/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/go/src/github.com/openshift/ci-operator/vendor/k8s.io/client-go/tools/record/event.go:226 +0x6e

goroutine 25 [IO wait, 8 minutes]:
internal/poll.runtime_pollWait(0x7f306e935f00, 0x72, 0xc42008b850)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420220618, 0x72, 0xffffffffffffff00, 0x1287ce0, 0x193b5d8)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc420220618, 0xc420774000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420220600, 0xc420774000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc420220600, 0xc420774000, 0x8000, 0x8000, 0x0, 0x8, 0x7ffb)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc42000e008, 0xc420774000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6a
crypto/tls.(*block).readFromUntil(0xc42023c090, 0x7f306e94c338, 0xc42000e008, 0x5, 0xc42000e008, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:493 +0x96
crypto/tls.(*Conn).readRecord(0xc420436000, 0x121b417, 0xc420436120, 0x20)
	/usr/local/go/src/crypto/tls/conn.go:595 +0xe0
crypto/tls.(*Conn).Read(0xc420436000, 0xc420417000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1156 +0x100
bufio.(*Reader).Read(0xc4200a6360, 0xc4203ea2d8, 0x9, 0x9, 0xc42008bcc0, 0x42a52b, 0x18)
	/usr/local/go/src/bufio/bufio.go:216 +0x238
io.ReadAtLeast(0x1285260, 0xc4200a6360, 0xc4203ea2d8, 0x9, 0x9, 0x9, 0xc4203a44e0, 0x43ccd1, 0x32)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x1285260, 0xc4200a6360, 0xc4203ea2d8, 0x9, 0x9, 0x121b468, 0xc42008bd10, 0x46ae63)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/openshift/ci-operator/vendor/golang.org/x/net/http2.readFrameHeader(0xc4203ea2d8, 0x9, 0x9, 0x1285260, 0xc4200a6360, 0x0, 0xc400000000, 0x80a460, 0xc420142528)
	/go/src/github.com/openshift/ci-operator/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/openshift/ci-operator/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203ea2a0, 0xc4201f2000, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/ci-operator/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/openshift/ci-operator/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run(0xc42008bfb0, 0x121a550, 0xc4204677b0)
	/go/src/github.com/openshift/ci-operator/vendor/golang.org/x/net/http2/transport.go:1428 +0x8e
github.com/openshift/ci-operator/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0xc4200fa4e0)
	/go/src/github.com/openshift/ci-operator/vendor/golang.org/x/net/http2/transport.go:1354 +0x76
created by github.com/openshift/ci-operator/vendor/golang.org/x/net/http2.(*Transport).newClientConn
	/go/src/github.com/openshift/ci-operator/vendor/golang.org/x/net/http2/transport.go:579 +0x651

goroutine 31 [chan receive, 131 minutes]:
github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc4200545c0)
	/go/src/github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/watch/mux.go:208 +0x57
created by github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/watch.NewBroadcaster
	/go/src/github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/watch/mux.go:75 +0xd5

goroutine 114 [semacquire, 131 minutes]:
sync.runtime_Semacquire(0xc4206a4ecc)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc4206a4ec0)
	/usr/local/go/src/sync/waitgroup.go:129 +0x72
github.com/openshift/ci-operator/pkg/steps.Run.func1(0xc4206a4ec0, 0xc4206da8a0)
	/go/src/github.com/openshift/ci-operator/pkg/steps/run.go:28 +0x2b
created by github.com/openshift/ci-operator/pkg/steps.Run
	/go/src/github.com/openshift/ci-operator/pkg/steps/run.go:27 +0x10a

goroutine 132 [chan receive, 8 minutes]:
github.com/openshift/ci-operator/pkg/steps.waitForBuildOrTimeout(0x12893e0, 0xc42043e3e0, 0xc4206907e0, 0xe, 0x118685f, 0x3, 0xa1dd00, 0x0, 0x0)
	/go/src/github.com/openshift/ci-operator/pkg/steps/source.go:278 +0x5ba
github.com/openshift/ci-operator/pkg/steps.waitForBuild(0x12893e0, 0xc42043e3e0, 0xc4206907e0, 0xe, 0x118685f, 0x3, 0x7f306e9916c8, 0x0)
	/go/src/github.com/openshift/ci-operator/pkg/steps/source.go:227 +0x6a
github.com/openshift/ci-operator/pkg/steps.handleBuild(0x12893e0, 0xc42043e3e0, 0xc420732000, 0x0, 0x1, 0xc42028c4b0)
	/go/src/github.com/openshift/ci-operator/pkg/steps/source.go:222 +0x26d
github.com/openshift/ci-operator/pkg/steps.(*sourceStep).Run(0xc420423ee0, 0x129fe80, 0xc4203a5c40, 0x100000200, 0x0, 0x0)
	/go/src/github.com/openshift/ci-operator/pkg/steps/source.go:114 +0x857
github.com/openshift/ci-operator/pkg/steps.runStep(0x129fe80, 0xc4203a5c40, 0xc4201f3170, 0xc4206da840, 0x0)
	/go/src/github.com/openshift/ci-operator/pkg/steps/run.go:114 +0x92
created by github.com/openshift/ci-operator/pkg/steps.Run
	/go/src/github.com/openshift/ci-operator/pkg/steps/run.go:76 +0x3a1

goroutine 129 [semacquire, 8 minutes]:
sync.runtime_notifyListWait(0xc420142540, 0xc400000002)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420142530)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/openshift/ci-operator/vendor/golang.org/x/net/http2.(*pipe).Read(0xc420142528, 0xc4205dd001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/ci-operator/vendor/golang.org/x/net/http2/pipe.go:64 +0x8f
github.com/openshift/ci-operator/vendor/golang.org/x/net/http2.transportResponseBody.Read(0xc420142500, 0xc4205dd001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/ci-operator/vendor/golang.org/x/net/http2/transport.go:1674 +0xa1
encoding/json.(*Decoder).refill(0xc4204c6780, 0x84d00a, 0x9)
	/usr/local/go/src/encoding/json/stream.go:159 +0x132
encoding/json.(*Decoder).readValue(0xc4204c6780, 0x0, 0x0, 0x103f3c0)
	/usr/local/go/src/encoding/json/stream.go:134 +0x23d
encoding/json.(*Decoder).Decode(0xc4204c6780, 0x1055dc0, 0xc420656160, 0x452190, 0xc42007b800)
	/usr/local/go/src/encoding/json/stream.go:63 +0x78
github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc4201f2ae0, 0xc420737300, 0x1000, 0x1300, 0xc4202d4500, 0x40, 0x38)
	/go/src/github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:150 +0x295
github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc42003a3c0, 0x0, 0x128d360, 0xc4202d4500, 0xa80, 0x12899e0, 0xc42082c298, 0x451fc0, 0xc42082c240)
	/go/src/github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x95
github.com/openshift/ci-operator/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc4206568e0, 0xc42059ffa8, 0x7, 0x12899e0, 0xc4204f2000, 0x0, 0x0)
	/go/src/github.com/openshift/ci-operator/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x7c
github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc4201f2b70)
	/go/src/github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93 +0x12e
created by github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/go/src/github.com/openshift/ci-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:60 +0xa8

rax    0xca
rbx    0x19b3f00
rcx    0xffffffffffffffff
rdx    0x0
rdi    0x19b4048
rsi    0x0
rbp    0x7ffc47e5d490
rsp    0x7ffc47e5d448
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc420468f68
r13    0xff
r14    0xff
r15    0xf
rip    0x459121
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

Duplicate manifests and runlevels not respected

Our manifests have this this input format (prefix 30):

30_00-namespace.yaml

but the output in origin-release is:

99_olm_30_00-namespace.yaml

I suspect this has to do with our ci-operator config: https://github.com/openshift/release/blob/master/ci-operator/config/operator-framework/operator-lifecycle-manager/master.yaml using the binary_build_commands (we also have three copies of our manifests in origin-release) but it’s not obvious to me how to fix it.

Detect when a config map doesn't exist on a running pod for > X time, fail

A common error mode is forgetting to create a dependent config map. We should do a slightly better job of conveying that condition back (either as a job failure or a log message)

build fails with invalid image reference

2018/07/27 08:21:05 Resolved source https://github.com/openshift/openshift-azure to master@ca1ec788, merging: #109 5c4dae7d @mjudeikis
2018/07/27 08:21:05 Resolved openshift/release:golang-1.10 to sha256:cfeddce041ed5ca39f531942853d6b0fbf7460f7d3edb4fbfda7ddeefedb10cd
2018/07/27 08:21:05 Resolved https://api.ci.openshift.org/azure/plugin-base:latest to sha256:872c657890a1d075378423d5322c8fc6829a02974e5eae5d75e2f397091567ad
2018/07/27 08:21:05 Running [input:root], [input:base], [release-inputs], src, bin, sync, [output:stable:sync], [images]
2018/07/27 08:21:05 Creating namespace ci-op-xd4b0blh
2018/07/27 08:21:05 Namespace will be deleted after 1h0m0s of idle time
2018/07/27 08:21:05 Tagging release images from azure/plugin-base:${component}
2018/07/27 08:21:05 Tagging openshift/release:golang-1.10 into pipeline:root
2018/07/27 08:21:05 Tagging https://api.ci.openshift.org/azure/plugin-base:latest into pipeline:base
2018/07/27 08:21:05 Building src
2018/07/27 08:21:37 Build src succeeded after 32s
2018/07/27 08:21:37 Building bin
2018/07/27 08:23:03 Build bin succeeded after 1m25s
2018/07/27 08:23:03 Building sync

$ oc get build sync -o yaml
apiVersion: build.openshift.io/v1
kind: Build
metadata:
  annotations:
    ci.openshift.io/job-spec: '{"type":"presubmit","job":"pull-ci-openshift-azure","buildid":"1","prowjobid":"c000afe8-9175-11e8-9196-0a58ac101a25","refs":{"org":"openshift","repo":"openshift-azure","base_ref":"master","base_sha":"ca1ec78862fb531c9fa26587ecf296e24204667a","pulls":[{"number":109,"author":"mjudeikis","sha":"5c4dae7df4543d52e44b35172203ba6572f70a90"}]}}'
  creationTimestamp: 2018-07-27T08:23:03Z
  labels:
    build-id: "1"
    created-by-ci: "true"
    creates: sync
    job: pull-ci-openshift-azure
    persists-between-builds: "false"
  name: sync
  namespace: ci-op-xd4b0blh
  ownerReferences:
  - apiVersion: image.openshift.io/v1
    controller: true
    kind: ImageStream
    name: pipeline
    uid: 01cc6bc5-9176-11e8-90c1-42010a8e0004
  resourceVersion: "35774515"
  selfLink: /apis/build.openshift.io/v1/namespaces/ci-op-xd4b0blh/builds/sync
  uid: 47d95a8c-9176-11e8-90c1-42010a8e0004
spec:
  nodeSelector: null
  output:
    to:
      kind: ImageStreamTag
      name: pipeline:sync
      namespace: ci-op-xd4b0blh
  postCommit: {}
  resources:
    limits:
      cpu: "2"
      memory: 4Gi
    requests:
      cpu: 100m
      memory: 200Mi
  serviceAccount: builder
  source:
    images:
    - as: null
      from:
        kind: ImageStreamTag
        name: pipeline:bin
      paths:
      - destinationDir: .
        sourcePath: /go/src/github.com/openshift/openshift-azure/sync
    - as:
      - sync
      from:
        kind: ImageStreamTag
        name: pipeline:root
      paths: null
    - as: null
      from:
        kind: ImageStreamTag
        name: pipeline:src
      paths:
      - destinationDir: .
        sourcePath: /go/src/github.com/openshift/openshift-azure///.
    type: Image
  strategy:
    dockerStrategy:
      dockerfilePath: Dockerfile.sync
      forcePull: true
      from:
        kind: ImageStreamTag
        name: pipeline:base
        namespace: ci-op-xd4b0blh
      imageOptimizationPolicy: SkipLayers
      noCache: true
    type: Docker
  triggeredBy: null
status:
  message: Referenced image could not be resolved.
  output: {}
  phase: New
  reason: InvalidImageReference

$ oc get all
NAME                          READY     STATUS      RESTARTS   AGE
pod/bin-build                 0/1       Completed   0          11m
pod/cleanup-when-idle-h487x   1/1       Running     0          11m
pod/src-build                 0/1       Completed   0          11m

NAME                          DESIRED   SUCCESSFUL   AGE
job.batch/cleanup-when-idle   1         0            11m

NAME                            TYPE      FROM         STATUS                        STARTED          DURATION
build.build.openshift.io/bin    Docker    Dockerfile   Complete                      11 minutes ago   1m25s
build.build.openshift.io/src    Docker    Dockerfile   Complete                      11 minutes ago   32s
build.build.openshift.io/sync   Docker                 New (InvalidImageReference)                    

NAME                                      DOCKER REPO                                             TAGS                       UPDATED
imagestream.image.openshift.io/pipeline   registry.svc.ci.openshift.org/ci-op-xd4b0blh/pipeline   bin,src,root + 1 more...   9 minutes ago
imagestream.image.openshift.io/stable     registry.svc.ci.openshift.org/ci-op-xd4b0blh/stable

$ oc logs build/src
Pulling image "registry.svc.ci.openshift.org/ci/clonerefs@sha256:7238ec8d8a2e35b177658405669d1823e01601aa0252cdb333b812549dce754a" ...

Pulling image docker-registry.default.svc:5000/ci-op-xd4b0blh/pipeline@sha256:cfeddce041ed5ca39f531942853d6b0fbf7460f7d3edb4fbfda7ddeefedb10cd ...
--> FROM docker-registry.default.svc:5000/ci-op-xd4b0blh/pipeline@sha256:cfeddce041ed5ca39f531942853d6b0fbf7460f7d3edb4fbfda7ddeefedb10cd as 0
--> ENV "CLONEREFS_OPTIONS"="{\"git_user_email\":\"[email protected]\",\"git_user_name\":\"ci-robot\",\"log\":\"/dev/null\",\"refs\":[{\"org\":\"openshift\",\"repo\":\"openshift-azure\",\"base_ref\":\"master\",\"base_sha\":\"ca1ec78862fb531c9fa26587ecf296e24204667a\",\"pulls\":[{\"number\":109,\"author\":\"mjudeikis\",\"sha\":\"5c4dae7df4543d52e44b35172203ba6572f70a90\"}],\"path_alias\":\"github.com/openshift/openshift-azure\"}],\"src_root\":\"/go\"}"
--> ADD ./clonerefs /clonerefs
--> RUN umask 0002 && /clonerefs && chmod g+xw -R /go/src
{"component":"clonerefs","level":"info","msg":"Cloning refs","refs":{"org":"openshift","repo":"openshift-azure","base_ref":"master","base_sha":"ca1ec78862fb531c9fa26587ecf296e24204667a","pulls":[{"number":109,"author":"mjudeikis","sha":"5c4dae7df4543d52e44b35172203ba6572f70a90"}],"path_alias":"github.com/openshift/openshift-azure"},"time":"2018-07-27T08:21:12Z"}
{"command":"os.MkdirAll(/go/src/github.com/openshift/openshift-azure, 0755)","component":"clonerefs","error":null,"level":"info","msg":"Ran command","output":"","time":"2018-07-27T08:21:12Z"}
{"command":"git init","component":"clonerefs","error":null,"level":"info","msg":"Ran command","output":"Initialized empty Git repository in /go/src/github.com/openshift/openshift-azure/.git/\n","time":"2018-07-27T08:21:12Z"}
{"command":"git config user.name ci-robot","component":"clonerefs","error":null,"level":"info","msg":"Ran command","output":"","time":"2018-07-27T08:21:12Z"}
{"command":"git config user.email [email protected]","component":"clonerefs","error":null,"level":"info","msg":"Ran command","output":"","time":"2018-07-27T08:21:12Z"}
{"command":"git fetch https://github.com/openshift/openshift-azure.git --tags --prune","component":"clonerefs","error":null,"level":"info","msg":"Ran command","output":"","time":"2018-07-27T08:21:13Z"}
{"command":"git fetch https://github.com/openshift/openshift-azure.git master","component":"clonerefs","error":null,"level":"info","msg":"Ran command","output":"From https://github.com/openshift/openshift-azure\n * branch            master     -\u003e FETCH_HEAD\n","time":"2018-07-27T08:21:14Z"}
{"command":"git checkout ca1ec78862fb531c9fa26587ecf296e24204667a","component":"clonerefs","error":null,"level":"info","msg":"Ran command","output":"Note: checking out 'ca1ec78862fb531c9fa26587ecf296e24204667a'.\n\nYou are in 'detached HEAD' state. You can look around, make experimental\nchanges and commit them, and you can discard any commits you make in this\nstate without impacting any branches by performing another checkout.\n\nIf you want to create a new branch to retain commits you create, you may\ndo so (now or later) by using -b with the checkout command again. Example:\n\n  git checkout -b new_branch_name\n\nHEAD is now at ca1ec78... moving house #2\n","time":"2018-07-27T08:21:15Z"}
{"command":"git branch --force master ca1ec78862fb531c9fa26587ecf296e24204667a","component":"clonerefs","error":null,"level":"info","msg":"Ran command","output":"","time":"2018-07-27T08:21:15Z"}
{"command":"git checkout master","component":"clonerefs","error":null,"level":"info","msg":"Ran command","output":"Switched to branch 'master'\n","time":"2018-07-27T08:21:15Z"}
{"command":"git fetch https://github.com/openshift/openshift-azure.git pull/109/head","component":"clonerefs","error":null,"level":"info","msg":"Ran command","output":"From https://github.com/openshift/openshift-azure\n * branch            refs/pull/109/head -\u003e FETCH_HEAD\n","time":"2018-07-27T08:21:16Z"}
{"command":"git merge 5c4dae7df4543d52e44b35172203ba6572f70a90","component":"clonerefs","error":null,"level":"info","msg":"Ran command","output":"Merge made by the 'recursive' strategy.\n pkg/addons/clean_test.go      | 222 ++++++++++++++++++++++++++++++++++++++++++\n pkg/validate/validate.go      |   6 --\n pkg/validate/validate_test.go | 116 ++++++++++++++++++++++\n 3 files changed, 338 insertions(+), 6 deletions(-)\n create mode 100644 pkg/addons/clean_test.go\n create mode 100644 pkg/validate/validate_test.go\n","time":"2018-07-27T08:21:16Z"}
{"component":"clonerefs","level":"info","msg":"Finished cloning refs","time":"2018-07-27T08:21:16Z"}
--> WORKDIR /go/src/github.com/openshift/openshift-azure/
--> RUN git submodule update --init
--> ENV "OPENSHIFT_BUILD_NAME"="src" "OPENSHIFT_BUILD_NAMESPACE"="ci-op-xd4b0blh"
--> LABEL "io.openshift.build.name"="src" "io.openshift.build.namespace"="ci-op-xd4b0blh"
--> Committing changes to temp.builder.openshift.io/ci-op-xd4b0blh/src:8605c1be ...
--> Done

Pushing image docker-registry.default.svc:5000/ci-op-xd4b0blh/pipeline:src ...
Pushed 2/3 layers, 70% complete
Pushed 3/3 layers, 100% complete
Push successful

$ oc logs build/bin
Pulling image docker-registry.default.svc:5000/ci-op-xd4b0blh/pipeline@sha256:6f230261fd6708d2129ef39e9118c0a89710132b99649c26705c9a7e0e732b03 ...
Pulled 2/3 layers, 71% complete
Pulled 3/3 layers, 100% complete
Extracting
--> FROM docker-registry.default.svc:5000/ci-op-xd4b0blh/pipeline@sha256:6f230261fd6708d2129ef39e9118c0a89710132b99649c26705c9a7e0e732b03 as 0
--> RUN ["/bin/bash","-c","set -o errexit; umask 0002; make sync"]
rm -f sync
go generate ./...
bindata.go
bindata.go
CGO_ENABLED=0 go build ./cmd/sync
--> ENV "OPENSHIFT_BUILD_NAME"="bin" "OPENSHIFT_BUILD_NAMESPACE"="ci-op-xd4b0blh"
--> LABEL "io.openshift.build.name"="bin" "io.openshift.build.namespace"="ci-op-xd4b0blh"
--> Committing changes to temp.builder.openshift.io/ci-op-xd4b0blh/bin:da482726 ...
--> Done

Pushing image docker-registry.default.svc:5000/ci-op-xd4b0blh/pipeline:bin ...
Pushed 3/4 layers, 78% complete
Pushed 4/4 layers, 100% complete
Push successful

not all pipeline tags available to templates through $LOCAL_IMAGE_*

We have use case where we need to pull few images as base images from other namespaces and use them in the template. Those images are heavy with a lot of tooling and static. So we pre-build them.

  "base_images": {
    "base": {
      "cluster": "https://api.ci.openshift.org",
      "namespace": "azure",
      "name": "plugin-base",
      "tag": "latest"
    },
    "origintest": {
      "cluster": "https://api.ci.openshift.org",
      "namespace": "azure",
      "name": "origin-test",
      "tag": "v3.10"
    }

Images got tagged to pipeline IS as inputs:

 Running [input:root], [input:base], [input:origintest], [release-inputs], src, bin, sync, verify, unit, [output:stable:sync], [images], cluster-launch-e2e-azure

If I try to resolve those in the template I get an error. Anything from ^ input:* is not available.

- name: IMAGE_ORIGINTEST
  required: true

The right way to get images from other namespaces would be

  "tag_specification": {
    "cluster": "https://api.ci.openshift.org",
    "namespace": "openshift",
    "name": "origin-v3.10",
    "tag": "",
    "tag_overrides": {}
  },

But we need to get images from 2 namespaces...

PR where this was tested:
openshift/release#1374

cc: @bbguimaraes @Kargakis
@openshift/sig-azure

RFE: stream pod logs from templates in the job log

At the moment, the ci-operator log is streamed in the job log view in prow. It would be nice to mux logs from test pods into that log.

Promotion of images is broken in 3.10

After #126 got merged, our postsubmit job that uses promotion broke because oc adm release does not exist in 3.10. We will probably stop using ci-operator:latest for now and rollback to a commit that makes promotion work again but ideally ci-operator should be intelligent enough not to run these commands in 3.10?
https://storage.googleapis.com/origin-ci-test/logs/branch-ci-openshift-openshift-azure/8/build-log.txt

2018/09/12 13:59:56 All images ready
2018/09/12 13:59:56 Create a new update payload image registry.svc.ci.openshift.org/ci-op-slyl46yq/release:latest
2018/09/12 13:59:56 Executing release release-latest
2018/09/12 14:00:12 Copying artifacts from release-latest into /logs/artifacts/release-latest
info: Using registry public hostname registry.svc.ci.openshift.org
Error: unknown flag: --max-per-registry


Usage:
  oc adm [flags]

Component Installation:
  router                             Install a router
  ipfailover                         Install an IP failover group to a set of nodes
  registry                           Install the integrated Docker registry

Security and Policy:
  new-project                        Create a new project
  policy                             Manage policy
  groups                             Manage groups
  ca                                 Manage certificates and keys
  certificate                        Modify certificate resources.

Node Management:
  create-node-config                 Create a configuration bundle for a node
  manage-node                        Manage nodes - list pods, evacuate, or mark ready
  cordon                             Mark node as unschedulable
  uncordon                           Mark node as schedulable
  drain                              Drain node in preparation for maintenance
  taint                              Update the taints on one or more nodes
  pod-network                        Manage pod network

Maintenance:
  diagnostics                        Diagnose common cluster problems
  prune                              Remove older versions of resources from the server
  build-chain                        Output the inputs and dependencies of your builds
  migrate                            Migrate data in the cluster
  top                                Show usage statistics of resources on the server
  verify-image-signature             Verify the image identity contained in the image signature

Configuration:
  create-kubeconfig                  Create a basic .kubeconfig file from client certs
  create-api-client-config           Create a config file for connecting to the server as a user
  create-bootstrap-project-template  Create a bootstrap project template
  create-bootstrap-policy-file       Create the default bootstrap policy
  create-login-template              Create a login template
  create-provider-selection-template Create a provider selection template
  create-error-template              Create an error page template

Other Commands:
  completion                         Output shell completion code for the specified shell (bash or zsh)
  config                             Change configuration files for the client

Use "oc adm <command> --help" for more information about a given command.
Use "oc adm options" for a list of global command-line options (applies to all commands).

2018/09/12 14:00:12 Container release in pod release-latest failed, exit code 1, reason Error

ci-operator config here
prowjob spec here

/kind bug

Graceful shutdown

Ensure graceful shutdown is done and we are not leaking clusters in e2e tests.

Move to dep from Glide

Currently this repo uses Glide for dependency management. It would be very good to move to dep.

Logging

We need a structured and verbose approach to logging so the ci-operator framework can be debugged after the fact.

Artifact location should not be surfaced to the end user

Why do we do this:

      "commands": "export ARTIFACT_DIR=/tmp/artifacts; ./test-backend.sh",
      "artifact_dir": "/tmp/artifacts"

The user does not care.

/cc @smarterclayton @droslean @petr-muller @bbguimaraes

Dry run mode

We need an entrypoint that will use the config to give an output api.List of items that will be created. We need to determine how to deal with fields that would otherwise be generated or dynamically resolved like the RPM server URL.

ci-operator does not support pulling multiple git repos

We want to create a job that tests Azure VM image builds. We have a playbook for building images in openshift-ansible.

The test flow is as follows:

run image node build playbook (openshift-ansible)
test built image is valid by using it to run a cluster (openshift-azure)
tag image and release

This job needs to pull openshift-ansible to do the node build in 1. (either via git for PRs and merges, or the published container image for periodic runs) AND openshift-azure for running the test deployment in 2.. We don't care about the commit pulled in 2.. At the moment, there is no support in ci-operator to make use of extra_refs which is not exposed via the downward API in prow. We are going to workaround this by putting openshift-azure in an image and use that directly in the template.

cc @openshift/sig-azure @stevekuznetsov @droslean @smarterclayton @bbguimaraes @petr-muller

On-boarding example format needs to be updated to YAML

https://github.com/openshift/ci-operator/blob/master/ONBOARD.md#using-separate-build-environment-and-release-environment-images

--give-pr-author-access-to-namespace=true broken

Added this option to the jenkins e2e ... from https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_jenkins/679/pull-ci-jenkins-e2e-canary/14/

2018/08/15 21:33:05 Resolved source https://github.com/openshift/jenkins to master@3b1ddf19, merging: #679 2c0e8fab @gabemontero
2018/08/15 21:33:05 Resolving inputs for the test
2018/08/15 21:33:05 Resolved openshift/release:golang-1.9 to sha256:5be0666e469c9ecd31f801b384aa4744426532110901cac978ad6754408c83fe
2018/08/15 21:33:05 Resolved openshift/origin-v3.11:cli to sha256:91e2531d9f97501bbed50d711cf87621808e03076d72946753a0b952f5d68155
2018/08/15 21:33:05 Resolved openshift/origin-v3.11:tests to sha256:b5c28a003c6b27ad171f1967c6a7c2c5eb58b3a5771f22aa177f0eab3b80bb1e
2018/08/15 21:33:05 Resolved inputs, targetting namespace ci-op-8qc1psnr
2018/08/15 21:33:05 Running [input:root], [input:base], [input:cluster-tests], [release-inputs], src, bin-2-centos, bin-slave-base-centos, jenkins-e2e-tests, prepare, tests, [images], bin-agent-maven-35-centos, bin-agent-nodejs-8-centos, maven_agent, nodejs_agent, e2e-gcp
2018/08/15 21:33:05 Creating namespace ci-op-8qc1psnr
2018/08/15 21:33:05 Creating rolebinding for user gabemontero in namespace ci-op-8qc1psnr
2018/08/15 21:33:05 Ran for 0s
error: could not initialize namespace: could not create role binding for: roles.rbac.authorization.k8s.io "admin" not found

@stevekuznetsov @droslean ptal