cbc02009 / ishioni-homelab-ops Goto Github PK

My Kubernetes Lab cluster ⛵️

... managed with Flux and Renovate 🤖

This is home to my personal Kubernetes lab cluster. Flux watches this Git repository and makes the changes to my cluster based on the manifests in the kubernetes directory. Renovate also watches this Git repository and creates pull requests when it finds updates to Docker images, Helm charts, and other dependencies.

There is a template over at onedr0p/flux-cluster-template if you wanted to try and follow along with some of the practices I use here.

My cluster is talos running on proxmox VMs. This is a semi hyper-converged cluster, workloads are sharing the same available resources on my nodes while I have a separate server for data storage.

🔸 Click here to see my Ansible playbooks and roles.

• cilium: Internal Kubernetes networking plugin
• cert-manager: Creates SSL certificates for services in my Kubernetes cluster
• external-dns: Automatically manages DNS records from my cluster in a cloud DNS provider
• external-secrets: Managed Kubernetes secrets using 1Password Connect
• k8s-gateway: Runs a separate internal-only DNS zone for some services
• ingress-nginx: Ingress controller to expose HTTP traffic to pods over DNS
• sops: Managed secrets for Kubernetes, Ansible and Terraform which are commited to Git
• Democratic CSI: Provides block and NFS storage provisioning
• volsync: Backup and recovery of persistent volume claims

Flux watches my kubernetes folder (see Directories below) and makes the changes to my cluster based on the YAML manifests.

Renovate watches my entire repository looking for dependency updates, when they are found a PR is automatically created. When some PRs are merged Flux applies the changes to my cluster.

This Git repository contains the following directories under kubernetes.

📁 kubernetes      # Kubernetes cluster defined as code
├─📁 bootstrap     # Flux installation
├─📁 flux           # Main Flux configuration of repository
└─📁 apps          # Apps deployed into my cluster grouped by namespace (see below)

While most of my infrastructure and workloads are selfhosted I do rely upon the cloud for certain key parts of my setup. This saves me from having to worry about two things. (1) Dealing with chicken/egg scenarios and (2) services I critically need whether my cluster is online or not.

The alternative solution to these two problems would be to host a Kubernetes cluster in the cloud and deploy applications like HCVault, Vaultwarden, ntfy, and Gatus. However, maintaining another cluster and monitoring another group of workloads is a lot more time and effort than I am willing to put in and only saves me roughly $18/month.

Over WAN, I have port forwarded ports 80 and 443 to the load balancer IP of my ingress controller that's running in my Kubernetes cluster. This is then managed by BGP to point to the right nodes.

dnsmasq is running on my USG Pro 4 and all DNS queries for internal domains are forwarded to k8s_gateway that is running in my cluster. With this setup k8s_gateway has direct access to my clusters ingresses and services and serves DNS for them in my internal network.

AdGuard DNS is used as the primary DNS for the whole network as I don't have the means to reliably run an adblocker selfhosted without making the setup unstable

external-dns is deployed in my cluster and configure to sync DNS records to Cloudflare. The only ingresses external-dns looks at to gather DNS records to put in Cloudflare are ones that I explicitly set an annotation of external-dns.home.arpa/enabled: "true"

🔸 Click here to see how else I manage Cloudflare with Terraform.

My home IP can change at any given time and in order to keep my WAN IP address up to date on Cloudflare. I have patched the vyatta part of unifi's vyos to work with cloudflare to make that work

Thanks to all the people who donate their time to the Kubernetes@Home Discord community. A lot of inspiration for my cluster comes from the people that have shared their clusters using the k8s-at-home GitHub topic. Be sure to check out the Kubernetes@Home search for ideas on how to deploy applications or get ideas on what you can deploy.

See the awful commit history

See LICENSE