The devkube from darshanime

devkube's Introduction

Bootstrap a Kubernetes cluster with custom binaries. No Kubeadm!

Devkube

Currently, a lot of Kubernetes bootstrappers are black boxes which don't provide a lot of flexibility on how the cluster is set up. Devkube changes that. Based off of Kelsey's KTHW, it wires together a cluster with custom binaries, complete with Flannel and CoreDNS.

Features

Use custom binaries for each component (kube-scheduler, kube-api-server, kubelet etc).
- The variables.yml file has links for the components.
- This is useful if you are developing a new Kubernetes feature and wish to test the change out on a real cluster
Change the flags on any component
- Update the config file for the component and run the relevant playbook to deploy the changes
Add new nodes
- Create the VM, run the relevant playbooks and the node should automatically join the cluster
Use cluster for testing manifests etc
- Since the cluster can be created in a few minutes on bare VMs, devkube can be used to quickly get clusters up and down for testing purposes

Getting started

Provision the VMs on your favorite cloud provider
- The VMs should be running Ubuntu 18.04 and be able to communicate with each other via private IPs (on all ports)
- The master node should have the 6443 port open for the api-server.
Download the required dependencies on your machine
- Ansible for running the playbooks
- cfssl for PKI
Edit the hosts.ini file
- Enter the public ip, the ssh-able username and the private ip of the server
- Make sure the server is ssh-able with the command ssh <user>@<public-ip> and the user has root access
- The first server (k8s-node-1) will become the master and all the other nodes will join in as workers
Run the play.yml playbook
- ansible-playbook play.yml -i hosts.ini

How does it work

Some of the main files are:

variables.yml
- This file has various top level configs, including release
- release has 2 possible values:
  - latest - this will pull in the latest artifact from kubernetes CI
  - v1.16.0-beta.1 etc - specific version, can be anything
root_certs.yml
- This sets up the root CA and generates the public and signing key for it
certs.yml
- This sets up all the certificates for the master and worker node components and transfers them to the servers
control_plane.yml
- This downloads the control plane binaries and bootstraps the control plane
workers.yml
- This downloads the worker node binaries and bootstraps the worker nodes.
- It also downloads installs flannel as the CNI network plugin and adds CoreDNS for DNS.

How do I...

Add a new node to the existing cluster
- Run certs.yml to generate the certs for the new node and transfer them to the server
- Run workers.yml to install the binaries and get the node up.
Change the flag on the api-server
- edit the kube-apiserver.service.j2 file and run the control_plane.yml playbook.
Change the kubelet binary
- update the variables.yml file with the new binary endpoint
- run workers.yml playbook
Avoid downloading worker node binaries
- this might be required if for example you change a flag on kubelet and don't need a fresh download of binaries
- run ansible-playbook workers.yml -i hosts.ini --skip-tags "downloads"

Contributing

Please file an issue if you face any problems. Better still, help fix it and make a PR!

devkube's People

Contributors

Stargazers

Watchers

devkube's Issues

support multiple masters

currently only k8s-node-1 is the master, create a master group and allow user speficied number of masters.

kube-scheduler failing on master without certs

Delete the master node, create a new one and run control_plane.yml, the kube-scheduler fails with this:

Jan 09 14:24:40 k8s-node-1 kube-scheduler[5391]: WARNING: the provided config file is an unsupported apiVersion ("componentconfig/v1alpha1"), which will be removed in future releases
Jan 09 14:24:40 k8s-node-1 kube-scheduler[5391]: WARNING: switch to command-line flags or update your config file apiVersion to "kubescheduler.config.k8s.io/v1alpha1"
Jan 09 14:24:40 k8s-node-1 kube-scheduler[5391]: WARNING: apiVersions at alpha-level are not guaranteed to be supported in future releases
Jan 09 14:24:40 k8s-node-1 kube-scheduler[5391]: invalid configuration: no configuration has been provided
Jan 09 14:24:40 k8s-node-1 systemd[1]: kube-scheduler.service: Main process exited, code=exited, status=1/FAILURE
Jan 09 14:24:40 k8s-node-1 systemd[1]: kube-scheduler.service: Failed with result 'exit-code'.

The kube-controller-manager fails too.

Jan 09 14:27:02 k8s-node-1 kube-controller-manager[6826]: invalid configuration: no configuration has been provided
Jan 09 14:27:02 k8s-node-1 systemd[1]: kube-controller-manager.service: Failed with result 'exit-code'.
Jan

Recommend Projects