If this plugin is missing a feature you need, consider sponsoring development. Pull requests always welcome!
A Dokku plugin to integrate with kubernetes.
- The
dokku-registryplugin should be installed and configured for your app - A configured kubectl (
/home/dokku/.kube/config) that can talk to your cluster - Dokku 0.20.4+
You can install this plugin by issuing the command:
dokku plugin:install https://github.com/dokku/dokku-scheduler-kubernetesAfter the plugin has successfully been installed you need to install the plugin's dependencies by running the command:
dokku plugin:install-dependenciesThe following functionality has been implemented
- Deployment and Service annotations
- Domain proxy support via the Nginx Ingress Controller
- Environment variables
- Letsencrypt SSL Certificate integration via CertManager
- Pod Disruption Budgets
- Resource limits and reservations (reservations == kubernetes requests)
- If no unit is specified, the values for memory are assumed to be in Mi (Megabytes)
- Zero-downtime deploys via Deployment healthchecks
- Traffic to non-web containers (via a configurable list)
Unsupported at this time:
- Custom docker-options (not applicable)
- Deployment timeouts (planned: #30)
- Dockerfile support (planned: #29)
- Encrypted environment variables (planned: #9)
- Proxy port integration (planned: #29)
- Manual SSL Certificates (planned: #28)
- The following scheduler commands are unimplemented:
If this plugin is missing a feature you need, consider sponsoring development. Pull requests always welcome!
- Each
Procfileentry will be turned into a kubernetesDeploymentobject. - Each
Procfileentry name must be a valid DNS subdomain. - The
webprocess will also create aServiceobject. - Non-web processes can create a
Serviceobject via a configurable property. - All created Kubernetes objects are tracked to completion via
kubedog. - All manifest templates are hardcoded in the plugin.
Set the scheduler to kubernetes. This can be done per-app or globally:
# globally
dokku scheduler:set --global selected kubernetes
# per-app
dokku scheduler:set $APP selected kubernetesYou also need to ensure your kubectl has the correct context specified:
# as the dokku user
kubectl config use-context YOUR_NAMEAnd configure your registry:
dokku registry:set $APP server gcr.io/dokku/Assuming your Dokku installation can push to the registry and your kubeconfig is valid, Dokku will deploy the app against the cluster.
The namespace in use for a particular app can be customized using the :set command. This will apply to all future invocations of the plugin, and will not modify any existing resources. If unspecified, the namespace in use is the cluster default namespace. The scheduler-kubernetes will create the namespace via a kubectl apply.
dokku scheduler-kubernetes:set $APP namespace testIf deploying from a private docker registry and the cluster needs does not have open access to the registry, an imagePullSecrets value can be specified. This will be injected into the kubernetes deployment spec at deploy time.
dokku scheduler-kubernetes:set $APP imagePullSecrets registry-credentialSee this doc for more details on creating an
imagePullSecretssecret file.
This functionality assumes a helm-installed
nginx-ingresscontroller:helm install ingress-nginx ingress-nginx/ingress-nginx --set controller.publishService.enabled=true
A Kubernetes Service object is created for each web process. Additionally, if the app has it's proxy-type set to nginx-ingress, then we will also create or update a Kubernetes ingress object within the namespace configured for the app. This can be set as follows:
dokku config:set $APP DOKKU_APP_PROXY_TYPE=nginx-ingressThe ingress object has the following properties:
- The name of the ingress object will be
app-ingress. - All kubernetes-deployed apps within the same namespace are added to the ingress object.
- Configured app domains are respected as unique rules.
- The configured service port for each rule is hardcoded to
5000.
To modify the manifest before it gets applied to the cluster, use the pre-kubernetes-ingress-apply plugin trigger.
Service objects can also be created for specific process types by configuring the service-process-types property. This is a comma-separated list that is specific to an individual application, and will always implicitly include the web process type.
dokku scheduler-kubernetes:set $APP service-process-types http,workerThe PORT environment variable is hardcoded to 5000. No Ingress object is created for non-web processes.
This functionality assumes a helm-installed
cert-managerCRD:kubectl create namespace cert-manager helm repo add jetstack https://charts.jetstack.io helm upgrade cert-manager jetstack/cert-manager --namespace cert-manager --version v1.10.0 --set installCRDs=true --install
At this time, the scheduler-kubernetes does not have support for custom SSL certificates. However, domains associated with an app can have a Letsencrypt SSL certificate provisioned automatically via the CertManager Kubernetes add-on.
To start using the CertManager, we will first need to set the issuer email
dokku config:set --global [email protected]Next, any apps that will require cert-manager integration will need to have that enabled:
dokku scheduler-kubernetes:set $APP cert-manager-enabled trueOn the next deploy or domain name change, the CertManager entry will be automatically updated to fetch an SSL certificate for all domains associated with all applications on the same ingress object.
A PodDisruptionBudget object can be created, and will apply to all process types in an app. To configure this, the pod-max-unavailable and pod-min-available properties can be set:
dokku scheduler-kubernetes:set $APP pod-min-available 1
# available in kubernetes 1.7+
dokku scheduler-kubernetes:set $APP pod-max-unavailable 1Pod Disruption Budgets will be updated on next deploy.
This feature requires an installed metric server, uses the
autoscaling/v2beta2api, and will apply immediately. Onlyresourcerules are supported when using the official metric-server, all others require the prometheus-operator and prometheus-adapter.
By default, Kubernetes deployments are not set to autoscale, but a HorizontalPodAutoscaler object can be managed for an app on a per-process type basis - referenced as $PROC_TYPE below. Using the HorizontalPodAutoscaler will disable the normal usage of ps:scale for the specified app/process-type combination, as per Kubernetes best practices.
At a minimum, both a min/max number of replicas must be set.
# set the min number of replicas
dokku scheduler-kubernetes:autoscale-set $APP $PROC_TYPE min-replicas 1
# set the max number of replicas
dokku scheduler-kubernetes:autoscale-set $APP $PROC_TYPE max-replicas 10You also need to add autoscaling rules. These can be managed via the :autoscale-rule-add command. Adding a rule for a target-name/metric-type combination that already exists will override any existing rules.
Rules can be added for the following metric types:
external:- format:
external:$NAME:$TYPE:$VALUE[:$SELECTOR] - fields:
$NAME: The name of the external metric to track$TYPE(valid values:[AverageValue, Value]): The type of the target.$VALUE: The value to track.$SELECTOR(optional): The selector to use for filtering to one or more specific metric series.
- format:
ingress:- format:
ingress:$NAME:$TYPE:$VALUE[:$INGRESS] - fields:
$NAME: The name of the ingress metric to track.$TYPE(valid values:[AverageValue, Value]): The type of the target.$VALUE: The value to track.$INGRESS(default:app-ingress): The name of the ingress object to filter on.
- format:
pods- format:
pods:$NAME:$TYPE:$VALUE - fields:
$NAME: The name of the metric from the pod resource to track.$TYPE(valid values:[AverageValue]): The type of the target.$VALUE: The value to track.
- format:
resource- format:
resource:$NAME:$TYPE:$VALUE - fields:
$NAME(valid values:[cpu, memory]): The name of the metric to track.$TYPE(valid values:[AverageValue, Utilization]): The type of the target.$VALUE: The value to track.
- format:
# set the cpu average utilization target
dokku scheduler-kubernetes:autoscale-rule-add $APP $PROC_TYPE resource:cpu:Utilization:50Rules can be listed via the autoscale-rule-list command:
dokku scheduler-kubernetes:autoscale-rule-list $APP $PROC_TYPERules can be removed via the :autoscale-rule-remove command. This command takes the same arguments as the autoscale-rule-add command, though the value is optional. If a rule matching the specified arguments does not exist, the command will still return 0.
# remove the cpu rule
dokku scheduler-kubernetes:autoscale-rule-remove $APP $PROC_TYPE resource:cpu:Utilization:50
# remove the cpu rule by prefix
dokku scheduler-kubernetes:autoscale-rule-remove $APP $PROC_TYPE resource:cpu:UtilizationAutoscaling rules are applied automatically during the next deploy, though may be immediately applied through the :autoscale-apply command:
dokku scheduler-kubernetes:autoscale-apply $APP $PROC_TYPEPods use pvcs as volumes. For volumes that support multiple access modes, the user specifies which mode is desired when using their claim as a volume in a Pod. See supported access modes by providers
# create a pvc
dokku scheduler-kubernetes:add-pvc $NAME $SIZE [--access-mode $MODE][--namespace $NAMESPACE ][--storage-class-name $CLASS]Fields:
- $NAME: The name of the persistent volume claim
- $SIZE is a numeric size of claim in MB.
- $MODE: Access mode must be either of ReadWriteOnce, ReadOnlyMany or ReadWriteMany. Default is ReadWriteOnce
- $NAMESPACE : The namespace for the pvc. Default is "default"
- $CLASS: The storage class name. Default is k8s providers default storage class
# list pvcs
dokku scheduler-kubernetes:list-pvc [$NAMESPACE]# delete pvc
dokku scheduler-kubernetes:remove-pvc $NAME --namespace $NAMESPACEPods access storage by using the claim as a volume.
# mounting a volume (requires a re-deploy to take effect)
dokku scheduler-kubernetes:mount $APP_NAME $PVC_NAME $CONTAINER_PATHFields:
- $APP_NAME: The name of the app
- $PVC_NAME: Name of persistent volume claim. Will be used as volume name. Claims must exist in the same namespace as the app.
List mounted volumes for an app:
# list mounted volumes
dokku scheduler-kubernetes:list-mount $APP_NAMEUnmount a volume:
# unmount a volume (requires a re-deploy to take effect)
dokku scheduler-kubernetes:unmount $APP_NAME $PVC_NAME $CONTAINER_PATHUnmount all volumes:
# unmount all (requires a re-deploy to take effect)
dokku scheduler-kubernetes:unmount-all $APP_NAMEWarning: Running this command exposes app environment variables to stdout.
The kubernetes manifest for a deployment or service can be displayed using the :show-manifest command. This manifest can be used to inspect what would be submitted to Kubernetes.
# show the deployment manifest for the `web` process type
dokku scheduler-kubernetes:show-manifest $APP $PROC_TYPE $MANIFEST_TYPEThis command can be used like so:
# show the deployment manifest for the `web` process type
dokku scheduler-kubernetes:show-manifest node-js-sample web
# implicitly specify the deployment manifest
dokku scheduler-kubernetes:show-manifest node-js-sample web deployment
# show the service manifest for the `web` process type
dokku scheduler-kubernetes:show-manifest node-js-sample web serviceThe command will exit non-zero if the specific manifest for the given app/process type combination is not found.
Warning: There is no validation for on annotation keys or values.
These can be managed by the :deployment-annotations-set command.
# command structure
dokku scheduler-kubernetes:deployment-annotations-set $APP $ANNOTATION_NAME $ANNOTATION_VALUE
# set example
dokku scheduler-kubernetes:deployment-annotations-set node-js-sample pod.kubernetes.io/lifetime 86400s
# unset example, leave the value empty
dokku scheduler-kubernetes:deployment-annotations-set node-js-sample pod.kubernetes.io/lifetimeCurrently, these apply globally to all processes within a deployed app.
These can be managed by the :pod-annotations-set command.
# command structure
dokku scheduler-kubernetes:pod-annotations-set $APP name value
# set example
dokku scheduler-kubernetes:pod-annotations-set node-js-sample pod.kubernetes.io/lifetime 86400s
# unset example, leave the value empty
dokku scheduler-kubernetes:pod-annotations-set node-js-sample pod.kubernetes.io/lifetimeCurrently, these apply globally to all processes within a deployed app.
These can be managed by the :service-annotations-set command.
# command structure
dokku scheduler-kubernetes:service-annotations-set $APP name value
# set example
dokku scheduler-kubernetes:service-annotations-set node-js-sample pod.kubernetes.io/lifetime 86400s
# unset example, leave the value empty
dokku scheduler-kubernetes:service-annotations-set node-js-sample pod.kubernetes.io/lifetimeCurrently, they are applied to the web process, which is the only process for which a Kubernetes Service is created.
These can be managed by the :ingress-annotations-set command.
# command structure
dokku scheduler-kubernetes:ingress-annotations-set $NAMESPACE name value
# set example
dokku scheduler-kubernetes:ingress-annotations-set my-namespace nginx.ingress.kubernetes.io/affinity cookie
# unset example, leave the value empty
dokku scheduler-kubernetes:ingress-annotations-set my-namespace nginx.ingress.kubernetes.io/affinityCurrently, these apply to all deployments within an namespace.
For deployments that use a rollingUpdate for rollouts, a rollingUpdate may be triggered at a later date via the :rolling-update command.
dokku scheduler-kubernetes:rolling-update $APPHealth checks for the app may be configured in app.json, based on Kubernetes
liveness and readiness
probes.
All Kubernetes options that can occur within a Probe
object
are supported, though syntax is JSON rather than YAML. The variable $APP may
be used to represent the app name.
If a process type is not configured for a given probe type (liveness or
readiness), any probe of the same type for the "*" default process type is
used instead.
Here (click the triangle to expand) is an example JSON for Kubernetes health checks.
{
"healthchecks": {
"web": {
"readiness": {
"httpGet": {
"path": "/{{ $APP }}/readiness_check",
"port": 5000
},
"initialDelaySeconds": 5,
"periodSeconds": 5
}
},
"*": {
"liveness": {
"exec": {
"command": ["/bin/pidof", "/start"]
},
"initialDelaySeconds": 5,
"periodSeconds": 5
},
"readiness": {
"httpGet": {
"path": "web processes override this.",
"port": 5000
},
"initialDelaySeconds": 5,
"periodSeconds": 5
}
}
}
}The following custom triggers are exposed by the plugin:
- Description: Allows a user to interact with the
deploymentmanifest after it has been submitted. - Invoked by:
- Arguments:
$APP$PROC_TYPE$MANIFEST_FILE$MANIFEST_TYPE - Example:
#!/usr/bin/env bash
set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
# TODO- Description: Allows a user to interact with the
ingressmanifest before it has been submitted. - Invoked by:
core-post-deploy,post-domains-update,post-proxy-ports-update, andproxy-build-configtriggers - Arguments:
$APP$MANIFEST_FILE - Example:
#!/usr/bin/env bash
set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
# TODO- Description: Allows a user to interact with the
deployment|servicemanifest before it has been submitted. - Invoked by:
scheduler-deploytrigger and:show-manifest - Arguments:
$APP$PROC_TYPE$MANIFEST_FILE$MANIFEST_TYPE - Example:
#!/usr/bin/env bash
set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
# TODO
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent