knative / test-infra Goto Github PK

Prow fails running any test on knative/build-templates#50

https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_build-templates/50/pull-knative-build-templates-unit-tests/1029758946673430529/

I0815 15:57:40.426] Call:  /workspace/./test-infra/jenkins/../scenarios/kubernetes_execute_bazel.py -- ./test/presubmit-tests.sh --unit-tests
W0815 15:57:40.449] Run: ('./test/presubmit-tests.sh', '--unit-tests')
W0815 15:57:40.547] Traceback (most recent call last):
W0815 15:57:40.547]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_execute_bazel.py", line 66, in <module>
W0815 15:57:40.547]     main(ARGS.cmd + ARGS.args)
W0815 15:57:40.548]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_execute_bazel.py", line 54, in main
W0815 15:57:40.548]     return_code = call(*cmd)
W0815 15:57:40.548]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_execute_bazel.py", line 40, in call
W0815 15:57:40.548]     return subprocess.call(cmd)
W0815 15:57:40.548]   File "/usr/lib/python2.7/subprocess.py", line 522, in call
W0815 15:57:40.548]     return Popen(*popenargs, **kwargs).wait()
W0815 15:57:40.548]   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
W0815 15:57:40.549]     errread, errwrite)
W0815 15:57:40.549]   File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
W0815 15:57:40.549]     raise child_exception
W0815 15:57:40.549] OSError: [Errno 2] No such file or directory
E0815 15:57:40.552] Command failed
I0815 15:57:40.552] process 648 exited with code 1 after 0.0m
E0815 15:57:40.553] FAIL: pull-knative-build-templates-unit-tests

Does the Prow config for the build-templates repo need to be updated?

/assign @adrcunha

The makefile only build is locally, but does not push it. we need to push it to the repo as well

/area test-and-release
/kind dev

Expected Behavior
When Prow job fails to run properly, we should get alert/notification automatically

Actual Behavior
We are finding out the failing job run manually.

This came up from scaling working group with @josephburnett.

PR owners often need to wait for approve/lgtm to close on a PR. However, from the reviewer side, they also have difficulty knowing when something needs their attention. It's normal for folks to get like a bazillion emails about Knative each day. We have an internal chrome pluggin for our google code review tool that gives me a little red flag in the corner of my browser when a cr needs my attention. something like that would be useful for knative. so individuals don't have to ping each person on slack.

Perhaps git notification can help. Let's discuss options.

Now that test-infra is stable, we should vendor the helper scripts it in the other repos and remove the fetching using docker from prow-tests image.

Internal Prow switched to timestamp-indexed builds, which cause resource names to be too long:

W0803 18:42:07.634] ERROR: (gcloud.container.clusters.create) ResponseError: code=400, message=cluster.name must be less than 40 characters.
W0803 18:42:07.682] 2018/08/03 18:42:07 process.go:152: Step 'gcloud container clusters create --quiet --enable-autoscaling --min-nodes=1 --max-nodes=3 --scopes=cloud-platform --project=gke-gerrit-boskos-02 --zone=us-central1-a --machine-type=n1-standard-4 --image-type=cos --num-nodes=3 --network=kbuild-templates-e2e-net1025451389398028288 --cluster-version=1.10.5-gke.3 kbuild-templates-e2e-cls1025451389398028288' finished in 657.926618ms
W0803 18:42:09.506] 2018/08/03 18:42:09 main.go:309: Something went wrong: starting e2e cluster: error creating cluster: error during gcloud container clusters create --quiet --enable-autoscaling --min-nodes=1 --max-nodes=3 --scopes=cloud-platform --project=gke-gerrit-boskos-02 --zone=us-central1-a --machine-type=n1-standard-4 --image-type=cos --num-nodes=3 --network=kbuild-templates-e2e-net1025451389398028288 --cluster-version=1.10.5-gke.3 kbuild-templates-e2e-cls1025451389398028288: exit status 1

Currently the testgrid bucket in gubernator is hardcoded, thus the version deployed is locally modified. Proper bucket definition (flag? yaml?) and deploy command (Makefile) needs to be done.

Running the update-codegen.sh script from eventing fails because PROW_JOB_ID is unbounded. This var should have a default automatically set when not running on prow.

$ ./hack/update-codegen.sh
./hack/../vendor/github.com/knative/test-infra/scripts/library.sh: line 32: PROW_JOB_ID: unbound variable

Sometimes you want multiple reviewers to weigh in before a change is merged, but Prow always merges as soon as the minimum set is satisfied (AFAIK).

Internally, we have the concept of WANT_LGTM to enumerate those you want an /lgtm from before merging a change. I feel like such a concept would be useful in Prow as well.

e.g. I'd like to write:

/want-lgtm @fejta
/want-lgtm @cjwagner

To have tide block merges until both have given the /lgtm, this should also implicitly /assign them.

I think it would also be useful to have /want-lgtm all that waits for /lgtm from all assigned reviewers.

cc @cjwagner @fejta @grantr @adrcunha @jessiezcc WDYT?

e.g., https://gubernator.knative.dev/pr/knative/serving/1833 shows no results.

Probably because GitHub access is not configured yet: https://gubernator.knative.dev/github_auth/pr

Expected

i have a kubernetes cluster that is bootstrapped by kubeadm, but currently, there is no way to let knative components run e2e on it

Not sure if this is something we configure to point to knative/test-infra
knative/pkg#97 (comment)

Git issue doesn't support priority. Would be nice to support P0, P1, P2 for issues opened.

• people can assign priority when open issue or during triage
• we can query issues with priority label

e.g. knative/serving#1977 lists no changed files in the log

This will simplify presubmit tests in the repos, unify features and increase coverage.

• Build tests helper should:

  • build all go code, plus any e2e tests;
  • check for 3rd party licenses and also up-to-date depencencies (if vendor present);
  • check go formatting;
  • check links and formatting of markdown files (meaning that build tests won't be totally skipped anymore for documentation-only PRs)

• Unit test helper should

  • run all unit tests (but NOT e2e tests) and report each test;
  • properly identify non-fatal errors that cause the test job to fail (but NO test to fail), like

  W0905 20:12:53.535] # github.com/knative/pkg/test
  W0905 20:12:53.535] test/e2e_flags.go:28:2: imported and not used: "github.com/knative/pkg/test/logging"
  

• Integration test helper should

  • run all E2E tests and report each test;
  • properly take care of the --emit-metrics flag, like

  local options=""
  (( EMIT_METRICS )) && options="--emit-metrics"
  ./test/e2e-tests.sh ${options}
  

If a contributor has an older version of dep there may be unnecessary edits to Gopkg.lock when opening a PR. Future PRs from other contributors with a newer version of dep may change the lock file. This thrashing is noisy.

It'd be nice if we could enforce a minimal version of dep as part of the presubmit checks.

Latest release at this time is v0.5.0
https://github.com/golang/dep/releases

https://testgrid.knative.dev/knative-pkg#coverage

The changes from #110 were pushed yesterday, and the other repos show coverage numbers for today. It seems that the target change caused the coverage report on knative/pkg to break.

The go coverage report only looks at files that have *_test.go files. This provides wrong coverage outputs and also bumps up the coverage numbers. It is not reflective of the current state.

go test does specify that the package has no test files and we should capture this behaviour in our report.

For eg. in knative/pkg/test, we have some go files but no _test files.
knative/pkg/test$ go test
? github.com/knative/pkg/test [no test files]

none of the jobs can be rerun with the command suggested by the re-run button

for example
kubectl create -f "https://prow.knative.dev/rerun?prowjob=e2aeff0e-99c2-11e8-80e3-0a580a3c0043"

Y'know, in case I'm hit by a bus :)

For example, in https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_eventing/276/pull-knative-eventing-build-tests/1027295713966428160/ there's a single error:

W0808 20:50:44.825] pkg/sources/github/receive_adapter/receive_adapter.go:20:2: imported and not used: "context"

but it's hidden in a long list of files, which makes it hard to spot.

The repo is now ready to be open, and this is a requirement for #30.

As we drive up code coverage numbers, # of tests (especially unit tests) increase significantly.
@evankanderson reported overall it's taking 30+min to run the tests, we should look at how much we can save if we run tests concurrently. go test has a parallel option that supports concurrency.

We can start with unit tests first.

as title suggests

When I visit: https://testgrid.knative.dev/knative-serving#continuous

Expected Behavior

No bot issues. :)

Actual Behavior

We've seen PRs where the bot complains that the CLA was not signed (even though it was before), like in knative/serving#1435; or where a PR created with pair programming like in knative/serving#1390 or knative/serving#1352.

Steps to Reproduce the Problem

1. Creating a PR with two different commiters is probably the easiest way to reproduce.

Additional Info

For knative/serving#1352, an adming was able to solve the problem by temporarily disabling and then re-enabling a setting about admins being able to edit PRs.

e.g., in https://gubernator.knative.dev/build/knative-prow/logs/ci-knative-serving-continuous/1039348282343559168 the e2e tests failed but were not reported

Although this is not Knative-specific, would be nice to get it fixed.

Gubernator contains several places where kubernetes is hardcoded. For example, when clicking "Ongoing" or "Incoming" in https://gubernator.knative.dev/pr you'll be taken to the GitHub search page with user hardcoded to kubernetes.

@adrcunha, can you pls document steps on how to onboard a new Knative repo to Knative Prow?

I hit an issue with the eventing e2e tests where the serving install fails because the Istio sidecar injector is not available to handle requests.

https://console.cloud.google.com/storage/browser/knative-prow/pr-logs/pull/knative_eventing/298/pull-knative-eventing-integration-tests/272/?angularJsUrl=%2Fstorage%2Fbrowser%2Fknative-prow%2Fpr-logs%2Fpull%2Fknative_eventing%2F296%2Fpull-knative-serving-integration-tests&authuser=1

Current approach using git frequently produces wrong results.

Now that we have prow-tests image, we can add a tool to it that would query GitHub about the changes and report the changes files in the current PR.

This was brought up from scaling working group.

"What about automatically notifying if LGTM is lost because of a merge?"

Since this will be our primary repo for productivity related projects, is it feasible to rename the repo to Productivity? "test-infra" is a bit limited in terms of scope.

In knative/build#335 the e2e tests report (https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_build/335/pull-knative-build-integration-tests/1039211002157076480/):

W0910 17:56:49.696] Fetching server config for us-central1
I0910 17:56:49.811] Latest GKE is 1.10.7, from [1.10.7-gke.1, /1.10.6-gke.3, /1.10.6-gke.2, /1.9.7-gke.6, /1.9.7-gke.5, /1.9.6-gke.2]

However, cluster creation fails with:

W0910 17:57:41.815] 2018/09/10 17:57:41 process.go:152: Step 'gcloud container clusters create --quiet --enable-autoscaling --min-nodes=1 --max-nodes=3 --scopes=cloud-platform --project=knative-boskos-05 --zone=us-central1-a --machine-type=n1-standard-4 --image-type=cos --num-nodes=3 --network=kbuild-e2e-net1039211002157076480 --cluster-version=1.10.7 kbuild-e2e-cls1039211002157076480' finished in 681.070848ms
W0910 17:57:42.751] 2018/09/10 17:57:42 main.go:309: Something went wrong: starting e2e cluster: error creating cluster: error during gcloud container clusters create --quiet --enable-autoscaling --min-nodes=1 --max-nodes=3 --scopes=cloud-platform --project=knative-boskos-05 --zone=us-central1-a --machine-type=n1-standard-4 --image-type=cos --num-nodes=3 --network=kbuild-e2e-net1039211002157076480 --cluster-version=1.10.7 kbuild-e2e-cls1039211002157076480: exit status 1

Now we have check_links_in_markdown function, I suggest to make it part of automatic presubmit check for all .md changes, so that no individual PR need to worry about add such tests.

For user trust reasons.

This way, working group triage will get the visibility to flaky tests and assign proper owners.
The bug should provide data points about flakiness per test. Prefer one issue per flaky test.

In https://gubernator.knative.dev/pr/knative_eventing/376, the links to the eventing repo are https://github.com/knative/knative_eventing, but they should be https://github.com/knative/eventing.

We need: google/go-containerregistry#253

This breaks some container analysis tooling, since some of the layer tarballs are malformed.

cc @imjasonh

In the prow logs, when creating a knative cluster with kubetest, we see:
util.go:132: Please use kubetest --gcp-service-account=/etc/service-account/service-account.json (instead of deprecated GOOGLE_APPLICATION_CREDENTIALS=/etc/service-account/service-account.json)

We should heed that warning and update our kubetest args

Root cause:

2018/08/07 08:28:18 error processing import paths in "config/webhook.yaml": no token in bearer response:
W0807 08:48:47.729] {"errors":[{"code":"DENIED","message":"Token exchange failed for project 'knative-releases'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control"}]}

Although this is not Knative-specific, would be nice to get it fixed.

Prow contains several places where kubernetes is hardcoded. For example, the default reply from Prow robot about an approval contains links to Kubernetes documentation like https://go.k8s.io/bot-commands and https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process. See #73 (comment) as an example.

kind (kubernetes in docker) might make e2e tests faster since there's no need to create an external cluster. However, it needs to be taken into consideration that we have to start a k8s cluster with a known, public version (e.g. 1.11.1).

When I open code coverage issues for each repo, I wish we can clone an existing issue, and just need to do minor updates to create a new issue.

/area test-and-release
/kind dev

Expected Behavior

There is no reason folks should have to build and push the e2e images and conformance test images whenever they want to run the tests, the images hardly ever change and they don't do much. They should be available publicly so folks can get up and running with the tests faster, and they should be regularly updated.

Note this will have to play nicely with a person's $DOCKER_REPO_OVERRIDE.

Actual Behavior

The tests expect anyone running the e2e or conformance tests to have built and pushed the images to their own repo.

Steps to Reproduce the Problem

n/a

Additional Info

Thanks @jonjohnsonjr !

This came up from discussion with Matt.
Number of (/test, /retest) is a good measurement for test flakiness. Let's include it as part of project health metric.

FlakinessSignal: Number of (/test, /retest) / Number of PR

Ideally, as number of PR grows, we'd like above ratio not to go up.

The e2e tests currently fail for me attempting to download a GKE release.

Will download and extract kubernetes-test.tar.gz from https://storage.googleapis.com/kubernetes-release-gke/release/v1.10.5-gke.3

Right now I can only run the e2e tests via prow for a PR. The e2e tests need to be runable without special privileges or infrastructure.