Travis services running on Kubernetes!

This repo contains configuration for all of the Kubernetes services that we run for Travis CI. Currently, we're only using this for our macOS-based infrastructure running on MacStadium.

This repo does not provision the actual Kubernetes cluster that the services run on. For that, see the terraform-config repo.

• travis-ci/terraform-config
• travis-ci/trvs-operator
• travis-ci/worker-operator

This repository is organized into three important directories:

• charts
• releases
• shared

The charts directory contains various Helm charts for the various services we deploy in Kubernetes. Each chart is parameterized so that it can be customized as needed and reused between environments.

If you need to make changes to how a service is deployed everywhere it is deployed, that change should go in the chart.

You can create a new chart using the helm create command:

$ helm create charts/my-new-service

The releases directory has subdirectories for each environment that we deploy to (e.g. macstadium-staging). The name should match up to the Terraform graph directory that provisions the cluster.

Inside a particular environment's releases directory are YAML files for the different Helm releases that will be deployed to the environment. Each chart deployed to an environment will have its own release where it can be customized as needed.

For example, to deploy jupiter-brain for travis-ci.org in the MacStadium staging environment, we have the following release resource:

apiVersion: helm.integrations.flux.weave.works/v1alpha2
kind: FluxHelmRelease
metadata:
  name: jupiter-brain-org
  namespace: macstadium-staging
  labels:
    chart: jupiter-brain
spec:
  chartGitPath: jupiter-brain
  releaseName: jupiter-brain-org
  values:
    secretEnv: staging-1

A few things to note:

• metadata.name should match the filename, and ideally should match spec.releaseName too.
• metadata.labels.chart must match spec.chartGitPath, and corresponds to the path to the chart relative to the charts/ directory.
• spec.values can and should be used to override the configuration of the chart for the particular release.
• releaseName should match the name of the chart, but may also include differentiators when the same chart is being deployed multiple times in the same namespace.

Deploying releases is done automatically in the cluster using Flux. No setup is needed on your local machine to do deploys.

However, you may want to inspect the state of the cluster from your machine. You can use the Kubernetes Dashboard for this, or you can set up a kubectl context to point at the right cluster and namespace.

You can configure such a context automatically from the terraform-config repo by running make context from the appropriate graph directory. This should only need to be done once per development machine, unless the cluster master has to be rebuilt for some reason.

Deploying changes is as easy as git pushing to master!

Existing environments should already have Flux set up (see shared/install-flux.sh for doing so in a new cluster). Flux as we use it comes in two pieces: flux and flux-helm-operator.

flux on its own just watches a Git repository, watches it for changes, and applies them to the cluster. In our clusters, flux is configured to only watch the releases/<env>/ directory for the corresponding environment. This means it will automatically update the FluxHelmRelease resources that define which Helm charts should be deployed in the cluster.

The flux-helm-operator does the rest of the work. It watches for changes in the FluxHelmRelease resources that are defined in the cluster and performs Helm upgrades of the corresponding charts as needed. It also watches for changes in the chart contents themselves.

Together, these two components ensure that what is in Git always matches what is running in the Kubernetes.

Our clusters are running kubernetes-dashboard, which makes it easy to what's going on in the cluster without having to run a bunch of kubectl commands.

• macstadium-prod-1
• macstadium-staging