Provide a mechanism to configure more than one Jenkins update center at a time.

Ensure Swarm User is properly created on Master with swarm password.

Ensure that all Jenkins masters have pip installed

When installing a Jenkins Master, some plugins are supposed to be installed, but they are not. Figure out why.

e.g. "Scriptler"

Running the development master environment on the CentOS 7 environments fails.

Steps to reproduce:

1. pip install -e .
2. cd vagrant/master
3. vagrant up
4. cinch hosts -e vagrant_dir=$(pwd) -e vagrant_provider=libvirt

Expected results:
Master provision completes successfully

Actual results:
Provisioning errors on step "TASK [jenkins_master : run Jenkins global config] ******************************"

Running cinch a second time completes successfully. Perhaps there is a status somewhere in the service that hasn't come up fully yet?

There are now beaker-client RPMs built into Fedora and EPEL7 repositories. Therefore, remove the beaker-project.org repository files from those groups and use the native packages, instead.

Right now, the output we get from Ansible is a big JSON blob that is not split on newlines, and is hard to read. Can we consider a way to split on newlines so that the output is easier to read? One implementation goes like so:

debug: var=service_swarm_err_output.stdout.split('\n')

Certainly this is not useful for all output, but should we spend some time to find a way to apply this globally?

https://github.com/willthames/ansible-lint

There are a few places where jenkins-cli.jar is used in our work. These should be eliminated in favor of directly calling methods on the host's API, where applicable, or directly modifying configuration and other files where that is the preferred method.

Since updating plugins requires a Jenkins restart, this task would need to be created in a way to inform the user that a restart would occur, and implement the necessary safety measures/warnings for a production Jenkins master.

Improve installing Jenkins plugins to directly call the API instead of passing calls through the Jenkins CLI jar file.

Personally I prefer a reverse proxy to deal with the fact that some webapps should not run on privileged ports. Production Jenkins servers will have an nginx reverse proxy, and in my personal opinion we should do the same for consistency. Note that this is a "consider" ticket and if anyone disagrees with my approach we can close this out after some discussion.

Create results and views in Jenkins Masters

Replace Apache SSL termination with Nginx for SSL termination.

Create a method to raise the configured ulimits for a Jenkins master

Configure Jenkins Master to listen on port 50000 for slaves, instead of relying on the default ports.

My playbook.yml:

• hosts: default
  vars:
  some certificates and repositories hidden
  gcc_compat_package: compat-gcc-44
  jenkins_rpm: jenkins
  jenkins_ssl_cert: cinch/vagrant/master_ssl/jenkins.crt
  jenkins_ssl_key: cinch/vagrant/master_ssl/jenkins.key
  jenkins_security_enabled: true
  roles:
  • certificate_authority
  • repositories
  • jenkins_common
  • jenkins_master

Test machine: centOs7
jenkins_rpm ~ jenkins-2.32

First error:
TASK [jenkins_master : download jenkins CLI] ***********************************
fatal: [default]: FAILED! => {"changed": false, "failed": true, "msg": "Destination /var/lib/jenkins not writable"}

Workaround with permissions 777

Errors about configuring users:

TASK [jenkins_master : configure CLI users appropriately] **********************
fatal: [default]: FAILED! => {"changed": false, "failed": true, "msg": "Roles not found - have you configured an admin using the Role-based Authorization Strategy?"}

Workaround with jenkins_security_enabled: false

Errors about plugins:
TASK [jenkins_master : install plugins] ****************************************
FAILED - RETRYING: TASK: jenkins_master : install plugins (3 retries left).
FAILED - RETRYING: TASK: jenkins_master : install plugins (2 retries left).
FAILED - RETRYING: TASK: jenkins_master : install plugins (1 retries left).
failed: [default] (item=build-name-setter) => {"attempts": 3, "details": "HTTP Error 403: Forbidden", "failed": true, "item": "build-name-setter", "msg": "Cannot get CSRF"}
FAILED - RETRYING: TASK: jenkins_master : install plugins (3 retries left).
FAILED - RETRYING: TASK: jenkins_master : install plugins (2 retries left).
FAILED - RETRYING: TASK: jenkins_master : install plugins (1 retries left).
failed: [default] (item=envinject) => {"attempts": 3, "details": "HTTP Error 403: Forbidden", "failed": true, "item": "envinject", "msg": "Cannot get CSRF"}
FAILED - RETRYING: TASK: jenkins_master : install plugins (3 retries left).
FAILED - RETRYING: TASK: jenkins_master : install plugins (2 retries left).
FAILED - RETRYING: TASK: jenkins_master : install plugins (1 retries left).
failed: [default] (item=greenballs) => {"attempts": 3, "details": "HTTP Error 403: Forbidden", "failed": true, "item": "greenballs", "msg": "Cannot get CSRF"}
FAILED - RETRYING: TASK: jenkins_master : install plugins (3 retries left).
FAILED - RETRYING: TASK: jenkins_master : install plugins (2 retries left).
FAILED - RETRYING: TASK: jenkins_master : install plugins (1 retries left).
failed: [default] (item=rebuild) => {"attempts": 3, "details": "HTTP Error 403: Forbidden", "failed": true, "item": "rebuild", "msg": "Cannot get CSRF"}

Lots of Ansible warnings are now coming out about our "when" lines with the release of Ansible 2.3. Mostly these warnings are straightforward to squelch by simply removing the Jinja2 templates from the line. By default, all "when" lines in a play are evaluated as a Jinja 2 expression, so adding those template lines is both unnecessary and now generates a warning.

Factor out the Beaker client work into its own role.

• Be sure that it includes repositories, until these are no longer necessary
• Be sure that it includes a dependency on system certificates
• Be sure it includes proper Beaker client configuration

Audit the command line options passed to the Jenkins service to be sure all needed values are being properly set.

linchpin init as of right now in the master branch creates a subdirectory named inventories, yet cinchpin expects the directory to be named inventory.

Ensure that Jenkins Service User is properly created and has SSH keys generated.

In the check_ssh role, we must run the verify that configured SSH private key is a file and has permissions of 0600 task and the check for SSH connectivity and authentication task on localhost. In 1e8ad1b the role was updated to be compliant with ansible-lint, but the removal of the local_action setting caused the check_ssh role to run remotely:

TASK [check_ssh : verify that configured SSH private key is a file and has permissions of 0600] ***
fatal: [10.8.180.33]: FAILED! => {"changed": false, "failed": true, "msg": "file (/home/vagrant/openstack-slave/keystore/mykey) is absent, cannot continue", "path": "/home/vagrant/openstack-slave/mykey", "state": "absent"}

I'm not sure why the remote host is used for this task (perhaps it's related to this ansible bug report), but I do know that we will need to revert the removal of local_action for now to ensure that the check_ssh role continues to work as expected.

If the Ansible variable jslave_name is set to something like 'cinch-slave', when the Jenkins slave is created it will had a UID of sorts appending to the end of the name, like 'cinch-slave-123fdsa'. We need to find out where this is coming from, and possibly make its existence user-configurable.

Ensure the iptables has the following rules:

• Accept all ICMP
• Accept all loopback
• Reject outgoing connections to 127.* that are not on loopback
• Allow inbound SSH
• Accept related and established connections
• Allow inbound TCP port 50000 for Jenkins slaves
• Allow inbound TCP port 8888 for ZeroMQ

Drop/deny everything else.

Now that linchpin supports postup and predestroy hooks as of the 1.0.0 release, we should try to use them. Here is an example of what hooks would look like in the linchpin PinFile:

    cinch:
      topology: cinch.yml
      layout: cinch.yml
      hooks:
        postup:
          - name: cinchup
            type: shell
            context: false
            actions:
              - echo Running postup
              - ansible-playbook -i inventories/cinch.inventory /paht/to/cinch/cinch/site.yml
        predestroy:
          - name: cinchdestroy
            type: shell
            context: false
            actions:
              - echo Running predestroy
              - ansible-playbook -i inventories/cinch.inventory /path/to/cinch/cinch/teardown.yml

Since we'll no longer need advanced functionality from iptables after moving to Nginx and we only use it as a firewall, I'd suggest moving the rules to firewalld for easier administration (adding new ports, etc.)

Regardless of the number of times it is run, the plugin installation process always reports at least some of the plug ins have been changed. Track down the reasons for this and mitigate, if possible (might require a PR to upstream Ansible to update the module - if this is the case, the patched version can live in our library/ folder until we upgrade to a version of Ansible that includes the fix).

Support a Jenkins Slave configuration that does nothing more than execute the latest Cinch-built Docker container directly on a host.

Create a method of running JJB on a configured git repository of jobs against configured Jenkins masters.

If the host is configured for LDAP/Kerberos, add the same set of authentication to Jenkins.

Also, provide a method to configure the host for LDAP/Kerberos authentication.

Ensure that SSH is installed and running on Jenkins masters.

Currently, in order for a user to add a custom plugin to their installation, they either have to edit the local code of their default.txt file or they have to fully duplicate the default.txt file contents into a host_var/group_var override of the jenkins_plugins variable and then add the extra plugins they want.

We should make default.txt truly the default list, and then provide a (default empty) list of extra plugins that can be installed, so the user does not have to duplicate or completely override the default.txt values.

In some cases, we want to pin a plugin to a specific version. The process is explained here: https://wiki.jenkins-ci.org/display/JENKINS/Pinned+Plugins . (Note: this is only applicable to Jenkins 1.X)

Given a list of plugin names
When a jenkins instance is deployed
Then each plugin name will have a file created in JENKINS_HOME/plugins/{plugin_name}.jpi.pinned

Example:
pinned_plugins=credentials,junit

will create:

• JENKINS_HOME/plugins/credentials.jpi.pinned
• JENKINS_HOME/plugins/junit.jpi.pinned

Ensure EPEL is configured in all RHEL and CentOS hosts.

This package has some benefits for interacting with cloud-init, such as specifying user data files for an OpenStack instance.

I've witnessed a case where the systemd swarm service started via Ansible, but later failed due to an authentication issue with the Jenkins master. In these cases we should watch the swarm service for some period of time to make sure it started successfully.

Jenkins masters should have the following packages installed:

1. gcc
2. python-devel
3. compat-gcc-34 (CentOS 6)
4. compat-gcc-44 (CentOS 7)
5. openssl-devel
6. libffi-devel
7. python-neutronclient
8. python-keystoneclient
9. python-glanceclient
10. python-novaclient

In RHEL and CentOS, this might require the RHOS repository to be activated.

After we upgrade to a version of linchpin that depends on Ansible >= 2.2.1 (currently we are pinned to >= 2.1), then the file cinch/library/jenkins_plugin.py should be dropped. The current version masks a buggy jenkins_plugin.py in Ansible 2.2.0 and absent from Ansible 2.1. Once we upgrade to 2.21 or later, we can drop the masking module.

Update provisioning to work when a slave is running in Fedora.

Also, update Docker builds to support a Fedora-based image.

Provide a method for loading private files into arbitrary places on the destination host. This will allow things such as keytab files, corporate SSL certificate keys, and the like to be loaded in without needing to be a part of the public repository.

Ensure that NTP is installed and operating on Jenkins masters

Ensure that Jenkins Master configuration matches version 1.3.0 of this Puppet module: https://forge.puppet.com/rtyler/jenkins/1.3.0/readme

Example: https://travis-ci.org/RedHatQE/cinch/jobs/231461426