After 3 commands a CI/CD pipeline with Jenkins, SonarQube, Nexus and a Docker Registry will be up and running on your Kubernetes cluster.
Follow this to install a microk8s kubernetes cluster on your laptop and optionally other nodes.
These are the 3 commands to run both the Jenkins master and the workers on a (possibly) distributed Kubernetes cluster.
git clone https://github.com/devops4me/docker-jenkins.git
cd docker-jenkins
kubectl apply -f jenkins-deployment.yaml
That's it. Your Kubernetes orchestrated Jenkins cluster is ready.
The sample jobs include
- a JAVA microservice and
- a JAVA library
kubectl get services -o wide
kubectl -n default logs -f deployment/jenkins --all-containers=true --since=5m
watch kubectl get pods -o wide
Visit the Jenkins UI with the IP address against the Jenkins service. Go to the jobs, click Build Now and then watch both the logs and the pods showing Kubernetes running dockerized pipelines on one or more nodes.
The dockerhub credentials will be used by Google Kaniko to push freshly built docker images into the a docker registry.
docker login
cat ~/.docker/config.json
kubectl create secret generic registrycreds \
--from-file=config.json=$HOME/.docker/config.json
docker logout
If using Dockerhub the config.json file will look something like this.
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "AzByCxxxxxxxxxxxxxxxxXcYbZa"
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/18.09.7 (linux)"
}
}Step 4 | Create safedb.net Git Credentials Kubernetes Secret
The safedb.net github credentials will be used by the ruby release gem instigated by the kubernetes Jenkins release pod to push the bumped up version number into the Git repository.
Go to a scratch folder on the machine and run these commands.
safe login <<book>>
safe open <<chapter>> <<verse>>
safe open github devops4me
safe write github.ssh.config # remove any extraneous config sections
safe write safedb.code.private.key
kubectl create secret generic safegitsshconfig \
--from-file=config=$PWD/config
rm config
kubectl create secret generic safegitsshkey \
--from-file=safedb.code.private.key.pem=$PWD/safedb.code.private.key.pem
rm safedb.code.private.key.pem
safe open rubygems.org do4me
safe write rubygems.org.credentials
kubectl create secret generic saferubygemscreds \
--from-file=credentials=$PWD/credentials
rm credentials
safe logout
kubectl get secret safegitsshconfig --output=yaml
kubectl get secret safegitsshkey --output=yaml
kubectl get secret saferubygemscreds --output=yaml
This is an example of a Pod's Yaml definition that uses the .ssh/config and the ssh private key created above.
metadata:
labels:
pod-type: jenkins-worker
spec:
containers:
- name: jnlp
env:
- name: CONTAINER_ENV_VAR
value: jnlp
- name: safehaven
image: devops4me/rubygem:latest
imagePullPolicy: Always
volumeMounts:
- name: gitsshconfig
mountPath: /root/gitsshconfig
- name: gitsshkey
mountPath: /root/gitsshkey
- name: gemcreds
mountPath: /root/gemcredentials
command:
- cat
tty: true
env:
- name: CONTAINER_ENV_VAR
value: safehaven
volumes:
- name: gitsshconfig
secret:
secretName: safegitsshconfig
- name: gitsshkey
secret:
secretName: safegitsshkey
defaultMode: 256
- name: gemcreds
secret:
secretName: saferubygemscreds
Note that
- we do not use
~because kubernetes does not understand it - we do not map to
/root/.sshas this folder will be read only and known hosts writing will fail - in the Jenkinsfile we copy
/root/gitsshconfig/configto the expected/root/.ssh/config - inside the config file we have specified the
/root/gitsshkey/<<private-key-name.pem>>location - we use
defaultMode: 256to set the0400file permissions for the key
Within the Jenkinsfile we need to take care of a number of issues.
stage('Release to RubyGems.org')
{
agent
{
kubernetes
{
yamlFile 'pod-image-release.yaml'
}
}
when { environment name: 'GIT_BRANCH', value: 'origin/master' }
steps
{
container('safehaven')
{
checkout scm
sh 'mkdir -p $HOME/.ssh && cp $HOME/gitsshconfig/config $HOME/.ssh/config'
sh 'mkdir -p $HOME/.gem && cp $HOME/gemcredentials/credentials $HOME/.gem/credentials'
sh 'chmod 0600 $HOME/.gem/credentials'
sh 'git config --global user.email [email protected]'
sh 'git config --global user.name "Apollo Akora"'
sh 'ssh -i $HOME/gitsshkey/safedb.code.private.key.pem -vT [email protected] || true'
sh 'git remote set-url --push origin [email protected]:devops4me/safedb.net.git'
sh 'git branch && git checkout master'
sh 'gem bump minor --tag --release --file=$PWD/lib/version.rb'
}
}
}
We verify the git push credentials within the safedb.net Jenkinsfile in the rubygem deploy stage right before the gem bump --release command.
ssh -i ~/.ssh/safedb.code.private.key.pem -vT [email protected]
If the ssh to github verification works - we can be confident that the kubernetes secrets have been correctly configured.
The safedb.net Jenkinsfile must configure the remote url for git push just before the git push is done.
git remote set-url --push origin [email protected]:devops4me/safedb.net.git
It is important for Jenkins jobs to complete safely without credentials as long as they are not in a given branch (usually master). This allows the team to run the jobs locally from development branches without a deployment happening.
Even when the ssh trial login command succeeds it still exits with return code 1 so we put || true to prevent it failing the build.
Step 7 | Create safedb.net RubyGem Credentials Kubernetes Secret
The safedb.net rubygem.org credentials will be used by the ruby release gem instigated by the kubernetes Jenkins release pod to deploy the packaged gem to the public gem repository.
docker login
cat ~/.docker/config.json
kubectl create secret generic registrycreds \
--from-file=config.json=$HOME/.docker/config.json
docker logout
If using Dockerhub the config.json file will look something like this.
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "AzByCxxxxxxxxxxxxxxxxXcYbZa"
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/18.09.7 (linux)"
}
}To view or change how the Kubernetes plugin configuration go to Jenkins | Manage Jenkins | Configure System and scroll down to the clouds section. The default configuration assumes
- the kubernetes API url is
https://kubernetes.default.svcfrom inside the cluster - the Jenkins master url the slaves connect to is
http://jenkinsas jenkins is the service name - the Jenkins master and slaves pods all run in the
defaultKubernetes namespace
This sample Java microservice shows how the Jenkinsfile and the pod-template yaml are written to fascilitate execution on a Kubernetes platform.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent