If the Forjfile given set org users and groups, the github plugin do not set it in github.yaml

Have:

users: {}
groups: {}

Should have:

users: {}
groups: 
  Core: 
    role: none
    users: [ "clarsonneur", "ojacques", "wenlock", "homeles", "arneluehrs", "miqui" ]
  Contributors:
    users: [ "vaceletm" ]
  gerrit:
    users: [ "forj-publish" ]

The pipeline defined by the jobdsl should provide a credential to avoid github quota to exceed.

We can have an existing organization which do not requires to be manage totally or partially by the github plugin.

So, the plugin should accept to not manage:

• organization users
• organization groups
• repositories, except the infra repository.

User story

forjj-jenkins and forjj-github are the new plugins repo.

We need to move related issues to respective repo.

This will improve transparency on what forjj-jenkins do.
ie, what and where are the source of jenkins generated files.

This will facilitate checking update changes

forjj-jenkins plugin should be able to configure properly the SSL layer for jenkins.

The jenkins process update is as follow:
http://sam.gleske.net/blog/engineering/2016/05/04/jenkins-with-ssl.html
and https://wiki.jenkins.io/display/JENKINS/Starting+and+Accessing+Jenkins

The key must be secured.

ssl can be set if we have the key, the SSL certificate and the CA certificate.

Should be teams-disabled and repos-disabled.

There is several reason why moving plugins in separated repos could make sense:

• Each plugin could have their own life cycle
• Each plugin could have their own Jenkinsfile/travis.yaml
• Each plugin manage their own list of issues
• Each plugin manages their own releases

But this will imply a different way to publish the plugin definition.
Today, we get the plugin.yaml from github. But I guess, it don't make sense to have a github flow for that.
We could deliver a service to push/pull those files as a simple directory forjj public service.

We need to think this in new version of forjj and plugins.

Find a way to maintain credentials stored in Jenkins.

The Jenkinsfile must be created, with basic forjj maintain and forjj maintain --emulate

When jenkins is started, jobdsl creates pipelines for all repos.

But even if the repo is not empty, jenkins do not see and re-index the pipeline, until you edit it and save it...

We need to find out why and fix it so that jenkins populate branches automatically.

forjj maintain return this everytime:

webhook 'blabla' updated

The API url is missing v3/in https://<Server>/api/v3

feature forj-oss/forjj#8

The plugin could be configured to support maintain webhook on repositories

This could be delivered by a webhooks section in Forjfile (ie new plugin global object)

If another plugin needs to set a webhook, we could imagine that forjj will set it as requested by the other plugin context.

This way we can fix the need to create the webhook to jenkins automaticall when both exist.
I don't konw right how this could be automated, but at least, this webhooks capabilty could help.

And in any case is useful in different other external (out of forjj control) integration.

Error found:

Processing DSL script ecloud-test-infra.groovy ERROR: invalid script name 'ecloud-test-infra.groovy; script names may only contain letters, digits and underscores, but may not start with a digit

2017/04/04 07:09:00 Running maintain on jenkins...
2017/04/04 07:09:02 environment checked.
2017/04/04 07:09:02 Maintaining 'jenkins'
2017/04/04 07:09:02 Running 'bin/build.sh && bin/start.sh'
2017/04/04 07:09:02 Using default Organisation/repo (forj-oss/jenkins-install-inits) for jenkins-install-inits. Add MYFORK= to change it.
2017/04/04 07:09:02 Using current git branch 'master'. Add BRANCH= to change it.
2017/04/04 07:09:02 + sudo -n docker pull forjdevops/jenkins-dood
2017/04/04 07:09:02 Using default tag: latest
2017/04/04 07:09:02 latest: Pulling from forjdevops/jenkins-dood
2017/04/04 07:09:02 Digest: sha256:ad5854dea942bcca2d32be63ad5ce2d7992d2e577ce99e1e52c463fd793ce8b2
2017/04/04 07:09:02 Status: Image is up to date for forjdevops/jenkins-dood:latest
2017/04/04 07:09:02 + sudo -n docker build -t hub.docker.com/devops/jenkins:test --build-arg JENKINS_INSTALL_INITS_URL=https://github.com/forj-oss/jenkins-install-inits/raw/master/ .
2017/04/04 07:09:02 Sending build context to Docker daemon 15.36 kB
2017/04/04 07:09:02 Step 1/8 : FROM forjdevops/jenkins-dood
2017/04/04 07:09:02  ---> 4ed57322637d
2017/04/04 07:09:02 Step 2/8 : COPY features.lst /tmp
2017/04/04 07:09:02  ---> Using cache
2017/04/04 07:09:02  ---> 9e90d31c2098
2017/04/04 07:09:02 Step 3/8 : ENV JENKINS_SLAVE_AGENT_PORT 50000
2017/04/04 07:09:02  ---> Using cache
2017/04/04 07:09:02  ---> 78f495025ce9
2017/04/04 07:09:02 Step 4/8 : USER root
2017/04/04 07:09:02  ---> Using cache
2017/04/04 07:09:02  ---> fee7c6d0d1ec
2017/04/04 07:09:02 Step 5/8 : ARG JENKINS_INSTALL_INITS_URL=https://github.com/forj-oss/jenkins-install-inits/raw/master/
2017/04/04 07:09:02  ---> Using cache
2017/04/04 07:09:02  ---> 71ee599543af
2017/04/04 07:09:02 Step 6/8 : RUN /usr/local/bin/jenkins-install.sh /tmp/features.lst
2017/04/04 07:09:02  ---> Using cache
2017/04/04 07:09:02  ---> c9375b990121
2017/04/04 07:09:02 Step 7/8 : RUN chown -R jenkins:jenkins $JENKINS_HOME /usr/share/jenkins/ref
2017/04/04 07:09:02  ---> Using cache
2017/04/04 07:09:02  ---> 967a1081c54a
2017/04/04 07:09:02 Step 8/8 : USER jenkins
2017/04/04 07:09:02  ---> Using cache
2017/04/04 07:09:02  ---> 07e89fb07606
2017/04/04 07:09:02 Successfully built 07e89fb07606
2017/04/04 07:09:02 + set +x
2017/04/04 07:09:02 + REPO=devops
2017/04/04 07:09:02 + IMAGE_NAME=jenkins
2017/04/04 07:09:02 + IMAGE_VERSION=test
2017/04/04 07:09:02 + [ /home/centos/infra/apps/ci/jenkins !=  ]
2017/04/04 07:09:02 + VOL_PWD=/home/centos/infra/apps/ci/jenkins
2017/04/04 07:09:02 + [  !=  ]
2017/04/04 07:09:02 + [  !=  ]
2017/04/04 07:09:02 + [ -f jenkins_credentials.sh ]
2017/04/04 07:09:02 + [ -f run_opts.sh ]
2017/04/04 07:09:02 + [ -f source_.sh ]
2017/04/04 07:09:02 + [  =  ]
2017/04/04 07:09:02 + echo SERVICE_ADDR not defined by any deployment environment. Set 'localhost'
2017/04/04 07:09:02 SERVICE_ADDR not defined by any deployment environment. Set 'localhost'
2017/04/04 07:09:02 + SERVICE_ADDR=localhost
2017/04/04 07:09:02 + [  =  ]
2017/04/04 07:09:02 + echo SERVICE_PORT not defined by any deployment environment. Set '8080'
2017/04/04 07:09:02 SERVICE_PORT not defined by any deployment environment. Set '8080'
2017/04/04 07:09:02 + SERVICE_PORT=8080
2017/04/04 07:09:02 + TAG_NAME=hub.docker.com/devops/jenkins:test
2017/04/04 07:09:02 + sudo docker ps -a -f name=jenkins-dood --format {{ .Image }}
2017/04/04 07:09:02 + CONTAINER_IMG=hub.docker.com/devops/jenkins:test
2017/04/04 07:09:02 + sudo docker images --format {{ .ID }} jenkins
2017/04/04 07:09:02 + IMAGE_ID=
2017/04/04 07:09:02 + [ hub.docker.com/devops/jenkins:test !=  ]
2017/04/04 07:09:02 + [ hub.docker.com/devops/jenkins:test != hub.docker.com/devops/jenkins:test ]
2017/04/04 07:09:02 + echo Nothing to re/start. Jenkins is still accessible at http://8080:8080
2017/04/04 07:09:02 Nothing to re/start. Jenkins is still accessible at http://8080:8080
2017/04/04 07:09:02 + exit 0
2017/04/04 07:09:02 
2017/04/04 07:09:02 -------------------------------------------

It refer to http://8080:8080 instead of http://:8080

Everytime jenkins is restarted, the container is removed. So any jobs and logs are simply lost.

We need to keep them after a restart for business continuity

Actually, with github, we use multibranch pipeline to create the pipeline.

Github organization provide a nice github integration (ui) in Jenkins.
If we want, we could ask forjj to use instead of mutlibranch pipeline as second option.

In this case, forjj needs to create a single organization dsl. It is possible to select which repository could be added. So, it will depends on needs.

This issue can be fixed manually. If you start it, the automation is back working.

So, the issue happen only when the job is freshly created by the job dsl code.

since jobs-dsl 1.60, a flag requires an admin to approve or refuse projects to get updated.

As projects update are managed through github flow, this feature could be interesting but break forjj to complete his automated task.

For forjj, the security approach about job-dsl must not be on the CI tool but on the SCM + team doing approval of code to be pushed.

So, we need forj-oss/jenkins-install-inits#15

Even if no update has to be provide, jenkins.yaml is written, but no commit message returned. So, forjj fails.

By default, jenkins plugin store jobdsl under jenkins instance code.

We should be able to choose a dedicated repo instead.

Be able to use or create a hubot service

It looks that Jenkins plugins need that to provide the link feedback and return feedback to github as well.

This is a requirement for github enterprise version.

forjj maintain should support for:

• forjj maintain --jenkins-shutdown
• forjj maintain --jenkins-safe-shutdown

Those 2 options will specifically request forjj jenkins plugin to restart jenkins itself.

We shoud never use this option in the infra jenkins job. Cutting the branch on which we are sitting...

If in case of create, a collection of team already exist, github must require a Force option to manage the list coming from Forjfile.
If github force is applied any unknown teams (not defined in Forjfile) will be simply removed.

In case of create, we may ask Forjj github to get the existing list of teams and manage it (Forjj will update the Forjfile, then)

if no teams has been identified in Forjfile, github should consider like teams-disabled.
Same for repo.

When pipeline is configured in jenkins, by default only org/repo is built.
If we want some recognized/trustable collaborator to gain on PR build, we need to add them at least with read access to the repo (even if the repo is public)

Currently this mist be done manually on the repo. Forjj should control this as well.

Could be with Name:

Ex:
repository: Test: Name: test-renamed
In normal case, name missed or equal to parent repo name, means not renamed.

Be able to use or create an hashicorp vault service for forjj credentials

The template.yaml could support features:

  common:
    - "feature:seed-job"
    - [...]
    - "{{ if .ProjectsHasSource 'github' }}plugin:github-branch-source:2.2.1{{ end }}"
    - "{{ if .ProjectsHasSource 'bitbucket' }}plugin:cloudbees-bitbucket-branch-source:2.2.2{{ end }}"
    - "{{ if .ProjectsHasSource 'git' }}plugin:git:3.4.1{{ end }}"

So, that we can add more features determines by source code repository given by proiects definition.

Today the github plugin requires the token to be created with appropriate rights.

When we create the Organization, we could ask for user/password to connect to github, generate the token and use it for all tasks as usual.

As soon as plugins has their own repo (Following #113 ), we should add their Jenkinfile