alexellis / k8s-on-raspbian Goto Github PK

Cluster Description:
Marcus Smallman' DIY Raspberry Pi Kubernetes Cluster

Related URL (Blog Post):
https://marcussmallman.io/2018/02/18/diy-rasberry-pi-kubernetes-cluster/

A while ago I was inspired to create my own cluster from reading a bunch of online resources. The prep.sh script in this repo was very helpful. I decided to write up my experience so that other can use/learn from.

Follow @davidcollom's comment in #16.

swap would be enabled after each boot: sudo systemctl disable dphys-swapfile followed by a reboot, fixed this.

Hello!

Thanks for sharing this nice guide. I have an idea, and I have some questions, maybe you can help me.
There is a project called YunoHost and it is aimed at helping home users self host services like WordPress or Nextcloud on an arm computer at home.
It is nice, but based on bash let's say, and as I love k8s api, I'm wondering about k8s mem footprint on such small devices.

In a context of a one node cluster, what is the mem footprint of:

• etcd ~ 22Mb I guess
• kube-api
• controller
• scheduler
• kubelet
  I think our main constraint here is mem and not cpu.

Another question, do we need also a network plugin? Or as there is only one node, it is enough?

And then, what about optimization, I think there are many rooms for improvements in this context:

• remove lot of unecessary code at compilation (like aws integration and so on)
• remove the scheduler, or replace it with a dummy one
• tune the controller for home user usage instead of thousands nodes, millions pods
• tune etcd
• reimplement some functions like cronjobs?
• socket activation for all the services

I'm just thinking out loud here, if you have any ideas, please share them here, and do not hesitate to close the issue once you answered!

Thanks again and have a nice day :)

When trying to follow the manual steps, I get a few errors, I hope these can be addressed.

When running this:

curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker
newgrp docker

I get this: newgrp: group 'docker' does not exist

Should it be??:

curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker \
newgrp docker

Anyway, if I try running the lines above, I get this: E: The repository 'https://download.docker.com/linux/raspbian 10 Release' does not have a Release file.

And when moving on in the steps, I get the same error later on when running:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
  echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
  sudo apt-get update -q && \
  sudo apt-get install -qy kubeadm

Any idea what this is? The is my first venture into kubernetes on Raspberry Pi, so it might just be me that's clueless here.

The current flannel yaml linked to by the guide (https://raw.githubusercontent.com/coreos/flannel/c5d10c8/Documentation/kube-flannel.yml) no longer starts the flannel pods due to updates since 1.10. Tolerations were added to the yaml in flannel-io/flannel#1044.

Expected Behaviour

Flannel pods should automatically start with the kubectl apply command

Current Behaviour

Flannel daemonsets are created but no pods are started

Possible Solution

Steps to Reproduce (for bugs)

1. Follow instructions to the install for a version >= 1.12.
2. For the networking step, take the flannel option
3. Check for flannel pods after kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Context

Nodes were stuck in NotReady state until I used the updated link.

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):

• What version of Kubernetes are you using? kubectl version: Server - 1.13.3 Client - 1.13.2

• Operating System and version (e.g. Linux, Windows, MacOS): Linux

In GUIDE.md, in the "Start up the Kubernetes dashboard" section, the kubectl apply ... command is referencing a resource that no longer exists. Here's the kubectl apply ... command from the guide along with the response:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/alternative/kubernetes-dashboard-arm-head.yaml
error: unable to read URL "https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/alternative/kubernetes-dashboard-arm-head.yaml", server reported 404 Not Found, status code=404

I poked around in the Kubernetes Dashboard repo and it appears that an ARM version is no longer available. I did find some deployment specs referencing older versions of the ARM dashboard image, but those images use the now retired Kubernetes Hipster instead of the Kubernetes Metrics Server. The dashboard pod log shows:
2020/03/07 17:00:00 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.

Expected Behaviour

I expected that the kubectl apply ... would successfully download the deployment spec and create the required resources.

Current Behaviour

See above, the kubectl apply ... returns a 404 not found

Possible Solution

Find a current arm version of the Kubernetes Dashboard.

Steps to Reproduce (for bugs)

1. Run the kubectl apply ... command referenced in GUIDE.md, i.e.,

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/alternative/kubernetes-dashboard-arm-head.yaml

Context

I was hoping to install the dashboard to visually inspect my cluster.

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):

$ d version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

• What version of Kubernetes are you using? kubectl version:

k version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-13T18:08:14Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/arm"}

• Operating System and version (e.g. Linux, Windows, MacOS):
  The client is a Mac

• What ARM or Raspberry Pi board are you using?
  Pi 4B

Following the guide in k8s-on-raspbian/GUIDE.md.
When I run:

pi@raspberrypi2:~ $ curl -4 http://127.0.0.1:31118 -d "test"

The HTTP request is sent but it is forever awaiting a response.

Environment

• Docker version 19.03.6
• Kubernetes v1.17.3
• Operating System: Raspbian 10 (buster)
• Board: PI4

All pods are running

pi@raspberrypi2:~ $ kubectl get pods --all-namespaces
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
default       markdownrender-f9744b577-gsmk9         1/1     Running   0          14m
kube-system   coredns-6955765f44-hsbhx               1/1     Running   0          23m
kube-system   coredns-6955765f44-zzf9w               1/1     Running   0          23m
kube-system   etcd-raspberrypi2                      1/1     Running   0          24m
kube-system   kube-apiserver-raspberrypi2            1/1     Running   0          24m
kube-system   kube-controller-manager-raspberrypi2   1/1     Running   0          24m
kube-system   kube-proxy-9zx54                       1/1     Running   0          11m
kube-system   kube-proxy-st75l                       1/1     Running   0          23m
kube-system   kube-scheduler-raspberrypi2            1/1     Running   0          24m
kube-system   weave-net-qhmps                        2/2     Running   2          11m
kube-system   weave-net-vhq6j                        2/2     Running   1          16m

Node (raspberrypi3) successfully joins

pi@raspberrypi2:~ $ kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
raspberrypi2   Ready    master   26m   v1.17.3
raspberrypi3   Ready    <none>   13m   v1.17.3

I'm using the right port:

pi@raspberrypi2:~ $ kubectl get service  markdownrender
NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
markdownrender   NodePort   10.106.161.23   <none>        8080:31118/TCP   17m

Routes show the 31118 port open

pi@raspberrypi2:~ $ netstat -tulpn
Proto Recv-Q Send-Q Local Address           Foreign Address         State     
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      
tcp        0      0 192.168.0.23:2379       0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      
tcp        0      0 192.168.0.23:2380       0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:2381          0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:45009         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:6784          0.0.0.0:*               LISTEN      
tcp6       0      0 :::10250                :::*                    LISTEN      
tcp6       0      0 :::10251                :::*                    LISTEN      
tcp6       0      0 :::6443                 :::*                    LISTEN      
tcp6       0      0 :::10252                :::*                    LISTEN      
tcp6       2      0 :::31118                :::*                    LISTEN      
tcp6       0      0 :::10256                :::*                    LISTEN      
tcp6       0      0 :::22                   :::*                    LISTEN      
tcp6       0      0 :::6781                 :::*                    LISTEN      
tcp6       0      0 :::6782                 :::*                    LISTEN      
tcp6       0      0 :::6783                 :::*                    LISTEN      

Comments

I believe I am missing something very simple. I already tried reset and init and the problem persists.

Expected Behaviour

sudo usermod pi -aG docker

should be sudo usermod pi -aG docker \

Current Behaviour

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):

• What version of Kubernetes are you using? kubectl version:

• Operating System and version (e.g. Linux, Windows, MacOS):

• What ARM or Raspberry Pi board are you using?

Hi, I follow the the instruction until the init step. At that point I am getting:

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.

Thanks for your help.

Context

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):
  sudo docker version
  Client:
  Version: 1.8.3
  API version: 1.20
  Go version: go1.4.3
  Git commit: f4bf5c7
  Built:
  OS/Arch: linux/arm

Server:
Version: 1.8.3
API version: 1.20
Go version: go1.4.3
Git commit: f4bf5c7
Built:
OS/Arch: linux/arm

• What version of Kubernetes are you using? kubectl version:
  sudo kubectl version
  Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/arm"}
  The connection to the server localhost:8080 was refused - did you specify the right host or port?

• Operating System and version (e.g. Linux, Windows, MacOS):

Mac OS High Sierra

• What ARM or Raspberry Pi board are you using?

Raspberry Pi 3 B+

When running sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16 it always fatally errors out

Expected Behaviour

This should complete the init

Current Behaviour

sudo kubeadm init --token-ttl=0 --pod-network-cidr=10.244.0.0/16
This results in a timeout and fatal errors.
Applying the timeout API patch does not help at all:
sudo sed -i 's/failureThreshold: 8/failureThreshold: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml && \ sudo sed -i 's/initialDelaySeconds: [0-9]\+/initialDelaySeconds: 360/' /etc/kubernetes/manifests/kube-apiserver.yaml

Context

I am setting up a new Raspberry Pi Master node.
This is repeatable, following this guide and always 100% results in this error.

Your Environment

R-Pi 4
SanDisk Extreme Pro 128GB
All steps in the guide - no need to list that here.

• What version of Kubernetes are you using? kubectl version:
  All steps in the guide - no need to list that here.

• Operating System and version (e.g. Linux, Windows, MacOS):
  Raspian Lite

• What ARM or Raspberry Pi board are you using?
  Raspberry Pi 4

Expected Behaviour

kubernetes master node starts and kubectl get pods shows Ready status.

Current Behaviour

kubernetes master node starts, and kubectl get pods shows NotReady status

kubectl describe nodes shows this error in the event log:
Failed to update Node Allocatable Limits ["kubepods"]: failed to set supported cgroup subsystems for cgroup [kubepods]: Failed to find subsystem mount for required subsystem: pids

Possible Solution

not sure - disable whatever requires this cgroup? is it something new in 1.14? or enable that cgroup in rasberian lite somewhere? (I'm not cgroup expert, so I don't know how to even start)

Steps to Reproduce (for bugs)

(follow the guide in this repo, I get these results at the "Check everything worked:" step of the guide)

Context

Can't schedule pods / nodes not ready.

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):
  Docker version 18.09.0, build 4d60db4

• What version of Kubernetes are you using? kubectl version:
  Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/arm"}
  Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/arm"}

• Operating System and version (e.g. Linux, Windows, MacOS):
  2018-11-13-raspbian-stretch-lite.img
  4.14.79-v7+

• What ARM or Raspberry Pi board are you using?
  Rasberry Pi 3 B+

This isn't an issue so much as a heads-up. Looks like this Medium article pretty much plagiarized the guide in this repo.

https://medium.com/developingnodes/setting-up-kubernetes-cluster-on-raspberry-pi-15cc44f404b5

Expected Behaviour

Get a dashboard up and running

Current Behaviour

The instruction to get a dashboard in pi cluster does not work.
The URL for alternate dashboard returns 404.

Possible Solution

Steps to Reproduce (for bugs)

1. Use Weave Net Network Driver
2. https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/ for this url for the dashboard setup.
3. Dashboard pod never starts

Context

Doing this for learning purposes. Wanted to see if the dashboard can be setup in an arm hardware.

Your Environment

Raspberry pi model 3 b+

• Docker version docker version (e.g. Docker 17.0.05 ):
  Docker version 18.09.0, build 4d60db4

• What version of Kubernetes are you using? kubectl version:
  kubectl version 1.13.1
  kubedm version 1.13.1
  kubelet version 1.13.1

• Operating System and version (e.g. Linux, Windows, MacOS):
  PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
  NAME="Raspbian GNU/Linux"
  VERSION_ID="9"
  VERSION="9 (stretch)"
  ID=raspbian
  ID_LIKE=debian
  HOME_URL="http://www.raspbian.org/"
  SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
  BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Expected Behaviour

seems to be a missing configuration, Could you please add it in the tutorial as a comment or a note please.

Current Behaviour

on raspbian, current version the /proc/sys/net/bridge/bridge-nf-call-iptables is not available

Possible Solution

sudo modprobe br_netfilter
with persistence
echo "br_netfilter" >> /etc/modules

Steps to Reproduce (for bugs)

1. Install a fresh rasbian lite in a PI3 b +

Your Environment

ARM PI 3 b+

• Docker version docker version (e.g. Docker 17.0.05 ):

• What version of Kubernetes are you using? kubectl version:
  Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/arm"}

• Operating System and version (e.g. Linux, Windows, MacOS): Linux

Readers' RPi clusters

Please comment here here with your name and URL to your cluster, plus any useful notes you have.

I'll then add it to the Readers' RPi clusters in the main README.

Alex

Join the #arm-and-pi channel on OpenFaaS Slack to chat with our community - https://docs.openfaas.com/community/