This tutorial demonstrates how to conduct distributed load testing using Kubernetes and includes a sample web application, Docker image, and Kubernetes controllers/services. For more background refer to the Distributed Load Testing Using Kubernetes solution paper.
- Google Cloud Platform account
- Install and setup Google Cloud SDK
Note: when installing the Google Cloud SDK you will need to enable the following additional components:
Compute Engine Command Line Interface
Developer Preview gcloud Commands
gcloud Alpha Commands
gcloud app Python Extensions
kubectl
Before continuing, you can also set your preferred zone and project:
$ gcloud config set compute/zone ZONE
$ gcloud config set project PROJECT-ID
First create the Google Container Engine clusters using the gcloud
command as shown below.
Note: This command defaults to creating a three node Kubernetes cluster (not counting the master) using the n1-standard-1
machine type. Refer to the gcloud alpha container clusters create
documentation information on specifying a different cluster configuration.
$ gcloud alpha container clusters create CLUSTER-NAME
After a few minutes, you'll have a working Kubernetes cluster with three nodes (not counting the Kubernetes master). Next, configure your system to use the kubectl
command:
$ kubectl config use-context gke_PROJECT-ID_ZONE_CLUSTER-NAME
Note: the output from the previous gcloud
cluster create command will contain the specific kubectl config
command to execute for your platform/project.
The webtest/bin
folder contains a simple Golang Web Application as the "system under test".
$ docker build -t USERNAME/webtest .
$ docker tag USERNAME/webtest gcr.io/PROJECT-ID/webtest
$ gcloud preview docker --project PROJECT-ID push gcr.io/PROJECT-ID/webtest
$ kubectl create -f webtest-rc.yaml
$ kubectl create -f webtest-service.yaml
Note: you will need the URL of the deployed sample web application when deploying the locust-master
and locust-worker
controllers.
$ kubectl get service webtest-service
First create the Google Container Engine clusters using the gcloud
command as shown below.
Note: This command defaults to creating a three node Kubernetes cluster (not counting the master) using the n1-standard-1
machine type. Refer to the gcloud alpha container clusters create
documentation information on specifying a different cluster configuration.
$ gcloud alpha container clusters create CLUSTER-NAME
After a few minutes, you'll have a working Kubernetes cluster with three nodes (not counting the Kubernetes master). Next, configure your system to use the kubectl
command:
$ kubectl config use-context gke_PROJECT-ID_ZONE_CLUSTER-NAME
Note: the output from the previous gcloud
cluster create command will contain the specific kubectl config
command to execute for your platform/project.
Before deploying the locust-master
and locust-worker
controllers, update each to point to the location of your deployed sample web application. Set the TARGET_HOST
environment variable in locust/kubernetes-config/locust-master-controller.yaml
and locust/kubernetes-config/locust-worker-controller.yaml
files.
The locust-master
and locust-worker
controllers are set to use the pre-built locust-tasks
Docker image, which has been uploaded to the Google Container Registry and is available at gcr.io/cloud-solutions-images/locust-tasks
. If you are interested in making changes and publishing a new Docker image, refer to the following steps.
First, install Docker on your platform. Once Docker is installed and you've made changes to the Dockerfile
, you can build, tag, and upload the image using the following steps:
$ docker build -t USERNAME/locust-tasks .
$ docker tag USERNAME/locust-tasks gcr.io/PROJECT-ID/locust-tasks
$ gcloud preview docker --project PROJECT-ID push gcr.io/PROJECT-ID/locust-tasks
Note: you are not required to use the Google Container Registry. If you'd like to publish your images to the Docker Hub please refer to the steps in Working with Docker Hub.
Once the Docker image has been rebuilt and uploaded to the registry you will need to edit the controllers with your new image location. Specifically, the spec.template.spec.containers.image
field in each controller controls which Docker image to use.
If you uploaded your Docker image to the Google Container Registry:
image: gcr.io/PROJECT-ID/locust-tasks:latest
If you uploaded your Docker image to the Docker Hub:
image: USERNAME/locust-tasks:latest
Note: the image location includes the latest
tag so that the image is pulled down every time a new Pod is launched. To use a Kubernetes-cached copy of the image, remove :latest
from the image location.
Now that kubectl
is setup, deploy the locust-master-controller
:
$ kubectl create -f locust-master-controller.yaml
To confirm that the Replication Controller and Pod are created, run the following:
$ kubectl get rc
$ kubectl get pods -l name=locust,role=master
Next, deploy the locust-master-service
:
$ kubectl create -f locust-master-service.yaml
This step will expose the Pod with an internal DNS name (locust-master
) and ports 8089
, 5557
, and 5558
. As part of this step, the type: LoadBalancer
directive in locust-master-service.yaml
will tell Google Container Engine to create a Google Compute Engine forwarding-rule from a publicly avaialble IP address to the locust-master
Pod. To view the newly created forwarding-rule, execute the following:
$ gcloud compute forwarding-rules list
Now deploy locust-worker-controller
:
$ kubectl create -f locust-worker-controller.yaml
The locust-worker-controller
is set to deploy 10 locust-worker
Pods, to confirm they were deployed run the following:
$ kubectl get pods -l name=locust,role=worker
To scale the number of locust-worker
Pods, issue a replication controller scale
command.
$ kubectl scale --replicas=20 replicationcontrollers locust-worker
To confirm that the Pods have launched and are ready, get the list of locust-worker
Pods:
$ kubectl get pods -l name=locust,role=worker
Note: depending on the desired number of locust-worker
Pods, the Kubernetes cluster may need to be launched with more than 3 compute engine nodes and may also need a machine type more powerful than n1-standard-1. Refer to the gcloud alpha container clusters create
documentation for more information.
The final step in deploying these controllers and services is to allow traffic from your publicly accessible forwarding-rule IP address to the appropriate Container Engine instances.
The only traffic we need to allow externally is to the Locust web interface, running on the locust-master
Pod at port 8089
. First, get the target tags for the nodes in your Kubernetes cluster using the output from kubectl get nodes
:
$ kubectl get nodes
NAME LABELS STATUS
gke-ws-0e365264-node-4pdw kubernetes.io/hostname=gke-ws-0e365264-node-4pdw Ready
gke-ws-0e365264-node-jdcz kubernetes.io/hostname=gke-ws-0e365264-node-jdcz Ready
gke-ws-0e365264-node-kp3d kubernetes.io/hostname=gke-ws-0e365264-node-kp3d Ready
The target tag is the node name prefix up to -node
and is formatted as gke-CLUSTER-NAME-[...]-node
. For example, if your node name is gke-mycluster-12345678-node-abcd
, the target tag would be gke-mycluster-12345678-node
.
Now to create the firewall rule, execute the following:
$ gcloud compute firewall-rules create FIREWALL-RULE-NAME --allow=tcp:8089 --target-tags gke-CLUSTER-NAME-[...]-node
To execute the Locust tests, navigate to the IP address of your forwarding-rule (see above) and port 8089
and enter the number of clients to spawn and the client hatch rate then start the simulation.
To teardown the workload simulation cluster, use the following steps. First, delete the Kubernetes cluster:
$ gcloud alpha container clusters delete CLUSTER-NAME
Next, delete the forwarding rule that forwards traffic into the cluster.
$ gcloud compute forwarding-rules delete FORWARDING-RULE-NAME
Finally, delete the firewall rule that allows incoming traffic to the cluster.
$ gcloud compute firewall-rules delete FIREWALL-RULE-NAME
To delete the sample web application, visit the Google Cloud Console.