The community continues to benefit from Kubernetes the Hard Way by Kelsey Hightower in understanding how each of the components work together and are configured in a reasonably secure manner, step-by-step. In a similar manner but using a slightly different approach, this guide attempts to demonstrate how the security-related settings inside kubernetes actually work from the ground up, one change at a time, validated by real attacks where possible.

By following this guide, you will configure one of the least secure clusters possible at the start. Each step will attempt to follow the pattern of a) educate, b) attack, c) harden, and d) verify in order of security importance and maturity. Upon completion of the guide, you will have successfully hacked your cluster several times over and now fully understand all the necessary configuration changes to prevent each one from happening.

The cluster built in this tutorial is not production ready--especially at the beginning--but the concepts learned are definitely applicable to your production clusters.

The target audience for this tutorial is someone who has a working knowledge of running a Kubernetes cluster (or has completed the kubernetes-the-hard-way tutorial) and wants to understand how each security-related setting works at a deep level.

• AWS EC2
• Ubuntu 16.0.4 LTS and search for '16.04 LTS hvm:ebs-ssd'
• Docker 1.13.x
• CNI Container Networking 0.6.0
• etcd 3.2.11
• Kubernetes 1.9.2

• AWS Account Credentials
  • Create/delete VPC (subnets, route tables, internet gateways)
  • Create/delete Cloudformation
  • Create/delete EC2 * (security groups, keypairs, instances)
• AWS cli tools configured to use said account
• bash
• git
• dig
• kubectl (v1.9.2)
• cfssl

• Etcd - t2.micro (10.1.0.5)
• Master - t2.small (10.1.0.10)
• Worker1 - t2.small (10.1.0.11)
• Worker2 - t2.small (10.1.0.12)

TODO

1. Create the VPC
2. Launch and configure the etcd instance
3. Launch and configure the master instance
4. Launch and configure the worker-1 and worker-2 instance
5. Create the local kubeconfig file

1. Deploy kube-dns
2. Deploy Heapster
3. Deploy Dashboard

1. Scan Ports
2. etcdctl
3. kubectl to API
4. curl to Kubelet/metrics

1. Security Groups
2. Etcd TLS
3. Cluster TLS
4. Admission Controllers

1. Helm/Tiller
2. Vulnapp
3. Azure Vote App

1. Multi-tenant Misuse

• Too many Pods
• Too many CPU/RAM shares

1. Separate Namespaces
2. Request/Limits on Pods
3. Namespace Resource Quotas
4. Metrics via Prometheus
5. Optional multi-etcd, mult-master

1. Malicious Image, Compromised Container, Multi-tenant Misuse

• Service Account Tokens
• Dashboard Access
• Tiller Access
• Kubelet Exploit
• Application Tampering
• Metrics Scraping
• Metadata API
• Outbound Scanning/pivoting

1. RBAC
2. New Dashboard
3. Separate Kubeconfigs per user
4. Tiller TLS
5. Kubelet Authn/z
6. Network Policy/CNI
7. Admission Controllers
8. Logging?

1. Malicious Image, Compromised Container, Multi-tenant Misuse

• Escape the container

1. Advanced admission controllers
2. Restrict images/sources
3. Network Egress filtering
4. Vuln scan images
5. Pod Security Policy
6. Encrypted etcd
7. Sysdig Falco

1. Delete Instances
2. Delete VPC

• Kubernetes the Hard Way - Kelsey Hightower