openshiftio / openshiftio-cico-jobs Goto Github PK

we should have the oc deploy commands wait for the deployment to finish and then have them report the status, into the same jenkins job that they were called from. Typically apps should go live/fail rapidly, so it wont add a lot more time to the overall job run

we need the oc deploy stage to be much smarter, and be ble to handle situations where a prev- deployment is still running. ref: https://ci.centos.org/view/Devtools/job/devtools-almighty-ui-build-master/10/console

almighty-ui is been moved to fabric8-planner. Now when we merge a PR to master, it is not deployed on demo.almighty. The the build system is looking at the old repos.

Please update the build system.

credential-id: 79f2b240-e87e-4216-ba88-6b3431bceb2f
https://gitlab.cee.redhat.com/dtsd/housekeeping/issues/1793

Details sent to @kbsingh

Our yaml file is growing more and more so we should start to organize it.

there are rate limits at 5 machines per 2 min and then 16 machines per 10 min in the centos ci infra per project, our jobs ignore the returned value from the 'cico node get' operation; we should fix this by adding a sleep 60; if the call fails and retry a few times before failing the job itself.

This wasent a problem in the past, since we only had 4 executors in jenkins, but with 8 executors, we are going to hit this a lot more frequently. specially when we have batch jobs coming up.

Since this script uses the internal libraries of JJB 1.x it is not compatible with JJB 2.x.

For example our Keycloak build downloads a lot of maven stuff. Some caching will help a lot to shorten the build time. And will help to avoid build failures in case of jboss nexus, etc. outages.

Is there a different service account that I could use to deploy to bayesian-preview project on rh-idev?

+ saasherder --environment staging update hash fabric8-analytics-pgbouncer 7fa39b22d4709f6832a9a3c5c82c3c923a0d2e37
INFO:saasherder.saasherder:Current context: bay
INFO:saasherder.saasherder:Service: fabric8-analytics-pgbouncer
INFO:saasherder.saasherder:Downloading: https://raw.githubusercontent.com/fabric8-analytics/fabric8-analytics-pgbouncer/7fa39b22d4709f6832a9a3c5c82c3c923a0d2e37/openshift/template.yaml
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com
INFO:saasherder.saasherder:Services written to file bay-services/pgbouncer.yaml.
+ saasherder --environment staging pull fabric8-analytics-pgbouncer
INFO:saasherder.saasherder:Current context: bay
INFO:saasherder.saasherder:Service: fabric8-analytics-pgbouncer
INFO:saasherder.saasherder:Downloading: https://raw.githubusercontent.com/fabric8-analytics/fabric8-analytics-pgbouncer/7fa39b22d4709f6832a9a3c5c82c3c923a0d2e37/openshift/template.yaml
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com
INFO:saasherder.saasherder:Template written to bay-templates/fabric8-analytics-pgbouncer.yaml
+ saasherder --environment staging template --output-dir fabric8-analytics-pgbouncer-processed tag fabric8-analytics-pgbouncer
INFO:saasherder.saasherder:Current context: bay
INFO:saasherder.saasherder:oc process  --output yaml -f bay-templates/fabric8-analytics-pgbouncer.yaml IMAGE_TAG=7fa39b2 > fabric8-analytics-pgbouncer-processed/fabric8-analytics-pgbouncer.yaml
+ oc apply -f fabric8-analytics-pgbouncer-processed/fabric8-analytics-pgbouncer.yaml -n bayesian-preview
Error from server: User "system:serviceaccount:dsaas-preview:platformdeployerbot" cannot "patch" "deploymentconfigs" with name "bayesian-pgbouncer" in project "bayesian-preview"
Error from server: User "system:serviceaccount:dsaas-preview:platformdeployerbot" cannot "patch" "services" with name "bayesian-pgbouncer" in project "bayesian-preview"
Build step 'Execute shell' marked build as failure
Finished: FAILURE

We realized that our builds completed successfully but the oc command was failing. The builds should fail when the oc command fails.

Example: https://ci.centos.org/job/devtools-almighty-core-build-master/530/console

This is related to PR #184.

This should harden our tests and find errors before the final merge.

As pointed out by @kbsingh , this is the correct line to add the merge:

https://github.com/almighty/almighty-jobs/blob/master/devtools-ci-index.yaml#L60

Images are check using check_image.py (from saasherder) in the promote-to-prod-test jobs. This validates the PRs.

However, we should also add this check to promote-to-prod tests, as there might be a lag between the PR and the merge.

In order to be able to support test scenarios where the github username != the OSIO username, we have to add a new variable/secret credential to the EE test.

At present, we have this variable defined: EE_TEST_USERNAME

What we need is an additional variable created: EE_TEST_GIT_USERNAME

See:
https://github.com/almighty/almighty-jobs/blob/master/devtools-ci-index.yaml#L842

Thanks!

( Just like the almighty API_URL) , the "stack" API url needs to be added as an environment variable for the fabric8-ui deployment.

@kbsingh

I think we can drop this job : https://ci.centos.org/view/Devtools/job/devtools-almighty-ui/ since the code under it has moved, and a new job has been created for -planner.

Can I get a +1 on that ?

Likely related to the fact that jobs on cico node end up in different folder on jenkins slave here

https://github.com/almighty/almighty-jobs/blob/6688e5599a76a8b546da9b07411923f84820cb09/devtools-ci-index.yaml#L1358

Yarn is a new package management system that was just released. It is significantly faster then NPM. I don't want to switch the UI to it until I know that the build system can handle it.

can the kompose tool be used to bringup adhoc app instances when PR builds succeed ?

we would need to limit it somewhere( at most 6 instances ? ), and the app would need to get reaped after a specified amount of time ( 12 hrs ? )

At the very least, we would need the kompose tool to create a route and allocate hostname to the UI and API ( ref: kubernetes/kompose#140 ), and we would need to setup a new account in openshift cluster that has a policy to terminate and reclaim all resources from apps after a specified amount of time.

cc: @kadel

At this moment, we are extracting the tag out of the git commit sha. However we cut it to 6 characters, we should instead follow the git rules for a short git sha (7 characters).

http://docs.openstack.org/infra/jenkins-job-builder/wrappers.html

http://sso.prod-preview.openshift.io/auth/realms/fabric8-test/broker/openshift-v3/endpoint

@alexeykazakov @hectorj2f :
Please fix the above url is it's not correct.

Our job is failing cause it is checking the master branch out, instead of the branch specified in the job https://github.com/almighty/almighty-jobs/blob/master/devtools-ci-index.yaml#L627.

https://ci.centos.org/job/devtools-keycloak-build-branch/1/console

We are going to need a way to monitor externally, the delta between master/ and the deployed content does not drift past a reasonable state ( ~ 6 to 8 hrs ? )

The following jobs are using more than 1G on the Jenkins master. Can we consider limiting the number of old builds we keep? artifacts.ci.centos.org is available if we need longer term storage of log files or built artifacts.

• devtools-fabric8-analytics-common-fabric8-analytics-pydoc
• devtools-fabric8-analytics-common-fabric8-analytics-pylint
• devtools-fabric8-analytics-server-fabric8-analytics
• devtools-fabric8-analytics-server-fabric8-analytics-pydoc
• devtools-fabric8-analytics-server-fabric8-analytics-pylint
• devtools-fabric8-analytics-worker-fabric8-analytics
• devtools-fabric8-analytics-worker-fabric8-analytics-pylint
• devtools-fabric8-auth
• devtools-fabric8-auth-coverage
• devtools-fabric8-planner-f8planner
• devtools-fabric8-ui
• devtools-fabric8-wit
• devtools-fabric8-wit-coverage

Would it make sense to rename this repository and move it under openshiftio org? Something like openshiftio/cico-jobs ?

@kbsingh

we had a situation that job https://ci.centos.org/job/devtools-e2e-fabric8-analytics/555/
was stuck for 39mins while timeout is set 30m https://github.com/openshiftio/openshiftio-cico-jobs/blob/master/devtools-ci-index.yaml#L3436

So I realized that not all templates have timeout applied like that https://github.com/openshiftio/openshiftio-cico-jobs/blob/master/devtools-ci-index.yaml#L313

this account should then be able to take ownership of the Dockerfile's and relationships with external resources like dockerhub etc.

We are currently using version 1.6.1, and the latest one is 2.0.10.

In 1.6.1 there is a bug that prevents the correct behaviour of a yaml that has:

- job:
    name: 'status-context'
    triggers:
      - github-pull-request:
          status-context: "status context"
          success-status: "success message"

The generated XML is invalid, and the status-context does not appear. This is solved in 2.0.10.

In order to reconfigure the new jobs with 2.0.10 we would need to start with a few ones, and make sure they continue to work and expand from there. This looks like a manual task.

we should get https://github.com/almighty/almighty-devdoc into the JJB definitions and get that tested in the same place as the -core and -ui components.