Tool to deploy a kubernetes cluster on openSUSE Kubic using kubeadm and salt
kubic-control consists of two binaries:
- kubicd, a daemon which communicates via gRPC with clients. It's setting up kubernetes on openSUSE Kubic, including pod network, kured, transactional-update, ...
- kubicctl, a cli interface
The communication is encrypted, the kubicctl command can run on any machine. The user authenticates with his certificate, using RBAC to determine if the user is allowed to call this function. kubiccd will use kubeadm and kubectl to deploy and manage the cluster. So the admin can at everytime modify the cluster with this commands, too, there is no hidden state-database except for the informations necessary for a kubernetes multi-master/HA setup.
Mainly generic requirements by kubernetes itself:
- All the nodes on the cluster must be on a the same network and be able to communicate directly with each other.
- All nodes in the cluster must be assigned static IP addresses. Using dynamically assigned IPs will break cluster functionality if the IP address changes.
- The Kubernetes master node(s) must have valid Fully-Qualified Domain Names (FQDNs), which can be resolved both by all other nodes and from other networks which need to access the cluster.
- Since Kubernetes mainly works with certificates and tokens, the time on all Nodes needs to be always in sync. Else communication inside the cluster will break.
Kubicd
/kubicctl
are using salt and certstrap to manage nodes and certificates, additional kubeadm, kubectl, kubelet and crio have to be installed.
Kubicd
has to run on the future kubernetes master node, kubicctl
can run anywhere on the network. This requires only that kubicd
is configured to listen on all interfaces, not only localhost
. The salt minions have to be already accepted on the salt master. Before kubicd
can be started, certificates have to be generated:
kubicctl certificates initialize
This will create a CA and several certificates in /etc/kubicd/pki
:
- Kubic-Control-CA.key - the private CA key
- Kubic-Control-CA.crt - the public CA key. This one is needed by kubicctl, too
- KubicD.key - the private key for kubicd
- kubicD.crt - the signed public key for kubicd
- admin.key - private key, allows kubicctl to connect to kubicd as admin
- admin.crt - public key, allows kubicctl to connect to kubcd as admin
For kubicctl
, you need to create a directory ~/.config/kubicctl
which
contains Kubic-Control-CA.crt
, user.key
and user.crt
. For the admin
role, this need to be a copy of admin.key and admin.crt. For other users,
you need to create corresponding certificates and sign them with
Kubic-Control-CA.crt
. If you call kubicctl
as root and there is no
user.crt
in ~/.config/kubicctl
, the admin certificates from
/etc/kubicd/pki
are used if they exist.
Certificates for additional users can be created with kubicctl certificates create <account>
.
Please take care of this certificates and store them secure, this are the passwords to access kubicd!
The first qustion is: single master node or high-availability kubernetes
masters? In the first case, there is not much to do: you need one master
machine, which is also running kubicd
, and that's it.
If you want a high-availability kubernetes master, you need three machines
which meet kubeadm's minimum requirements for masters. Additional, you need a
load balancer. The load balancer must be able to communicate with all control
plane nodes on the apiserver port. It must also allow incoming traffic on its
listening port 6443. If you have none, you can use HAProxy.
To deploy the control-plane on the master with weave as POD network and
kured
to manage the reboot of nodes:
kubicctl init
To deploy a high-availability master, the DNS name of the load balancer needs to be specified. The cluster will be reacheable under this DNS name.
kubicctl init --multi-master load.balancer.dns
If the haproxy is also a salt-minion and should be automatically configured and adjusted:
kubicctl init --haproxy salt-minion --multi-master load.balancer.dns
In this case, kubicd will configure the haproxy and add or remove master nodes
depeding on the kubernetes cluster configuration automatically, if haproxycfg
is installed.
For cilium or flannel instead of weave you have to use kubicctl init --pod-network cilium
or kubicctl init --pod-network cilium
.
To add additional worker nodes:
kubicctl node add node1,...
In the high-availability case, two additional masters needs to be added:
kubicctl node add --type master master2
kubicctl node add --type master master3
Make sure the load balancer can reach all three master nodes.
In the same way as new nodes were added, existing nodes can also be removed:
kubicctl node remove
or reboot nodes: kubicctl node reboot
. Please make
sure that you have always three master nodes in case of high-availbility master.
To access with cluster with kubectl
, you can get the kubeconfig with:
kubicctl kubeconfig
.
The kubernetes cluster can be upgraded with:
kubicctl upgrade
Kubicd
reads two configuration files: kubicd.conf
and rbac.conf
. The
first one is optional and contains the paths to the certificates and the server
name with port kubicd
should listen to. The default file can be found in
/usr/etc/kubicd/kubicd.conf
. The variables can be overriden with
/etc/kubicd/kubicd.conf
, which only contains the changed entries.
The second file, rbac.conf
, is mandatory, else nobody can access kubicd
,
all requests will be rejected. The default file can be found in
/usr/etc/kubicd/rbac.conf
. Changed entries should be written
to /etc/kubicd/rbac.conf
.
rbac.conf
contains the roles as key and the users, who are allowed to use
this functionality as comma seperated list. kubicctl rbac list
will print
out a list of current configured roles and the corresponding users. kubicctl rbac add <role> <user>
will add the user to the role.
kubicd
has support to deploy new nodes with help of
Yomi. This requires salt pillars for the
new node describing what should be where installed. There is a kubicctl node deploy prepare
command, which will create the required pillars based on the
type of the new node and the hardware. After this step, the generated pillars
should be verified, if really the right things will be deleted and installed.
Afterwards, kubicctl node deploy install
will erase the content of the
harddisk, install the new node and, if this new node is of type "master" or
"worker", will also add it to the kubernetes cluster.
- certificates - Manage certificates for kubicd/kubicctl communication
- create - Create certificate for an user. The certificate will be stored in the local directory where you did call kubicctl.
- initialize - Create CA, KubicD and admin certificates. This certificates will be stored in
/etc/kubicd/pki/
- help - Help about any command
- init - Initialize Kubernetes Master Node
- --multi-master= Setup HA masters, the argument must be the DNS name of the load balancer
- --haproxy= Adjust haproxy configuration for multi-master setup via salt
- --pod-network=<flannel|cilium> Pod network
- --adv-addr= IP address the API Server will advertise on
- --stage=<official|devel> Specify to use the official images or from the devel project
- kubeconfig - Download kubeconfig
- --output= - Where the kubeconfig file should be stored
- node - Manage kubernetes nodes
- add ,... - Add new nodes to cluster. Node names must be the name used by salt for that node. A comma seperated list or '[]' syntax are allowed to specify more than one new node.
- list - List all reacheable worker nodes
- reboot - Reboot node from cluster. Node will be drained first. Node name must be the name used by salt for that node.
- remove - Remove node form cluster
- deploy - Install a new node
- prepare - Prepare configuration to install new node with Yomi
- install - Install new node with Yomi
- rbac - Manage RBAC rules
- add - Add user account to a role
- list - List roles and accounts
- upgrade - Upgrade Kubernetes Cluster to the version of the installed kubeadm command if not otherwise specified
- destroy-cluster - Remove all worker and master nodes
- status - Print status informations of KubicD
- version - Print version information
On the machine where kubicd
is running, /etc/kubicd
and
/var/lib/kubic-control
should be part of the backup.
Kubicd
does not store any informations about the state of the kubernetes
cluster except for the deployed daemonsets. This allows to manage the cluster
with kubectl
and kubeadm
yourself without kubicctl
. Daemonsets not
installed via kubicctl
/kubicd
have to be updated by the admin themself,
they will not be updated by kubicctl upgrade
.
There is only one important thing: a grain has to be set on new worker and
master nodes: kubicd=kubic-worker-node
for worker,
kubicd=kubic-master-node
for additional master nodes. If nodes
gets manual removed, this grain has to be deleted, too.