Comments (22)
PIDs cgroup will be available on the next rpi release: raspberrypi/linux#2968 (comment).
from k8s-on-raspbian.
@Mike-Dunton I was able to rpi-update to 4.19.46-v7+ today, and confirm the PIDS fix is in place. kubeadm 1.14.3 installs and inits fine.
from k8s-on-raspbian.
I've managed to upgrade my raspbian to buster, but it wasn't error free, the following issues where hit:
iptables in nf_tables mode - kube-proxy only works in legacy mode kubernetes/kubernetes#71305 (comment)
swap would be enabled after each boot: sudo systemctl disable dphys-swapfile
followed by a reboot, fixed this.
I noticed a substantial amount of errors being reported from docker reporting that cgroupsfs/net_prio being missing (even though it existed and mounted) - upgrading docker-ce to 18.09.0
resolved these - in addition - I had to install docker from stretch repo as busters is known to be broken: docker/for-linux#709
from k8s-on-raspbian.
Recompiling Raspbian kernel with CONFIG_CGROUP_PIDS enabled fixed this issue for me (I was running k8s v1.14.1)
from k8s-on-raspbian.
Thanks hgontijo to push for it :)
from k8s-on-raspbian.
I had the same issue. Did a dist-upgrade
from stretch to buster and now kubernetes version v1.15.0
works. Buster has kernel version 4.19.50-v7+
and includes the PIDS Cgroup.
However there is a warning when running kubelet describe node master
that says kubelet is posting ready status. WARNING: CPU hardcapping unsupported
. I'm not sure if this bothers kubernetes but since it's running fine i guess not.
Note: After upgrading in place i had to disable swap again.
from k8s-on-raspbian.
@davidcollom Thank you for those hints. Didn't notice that iptables wasn't working correctly but fixed it now with the command from your link.
from k8s-on-raspbian.
I met the same issue on k8s v1.14.1
from k8s-on-raspbian.
Based on the error message, it looks to me a side effect of Pid limiting, which includes Pod Pids Limit and Node Pids Limit features introduced on v1.14.0, that requires the pids
cgroup.
The pids cgroups is not mounted on Raspbian:
$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 8 21 1
cpu 4 86 1
cpuacct 4 86 1
blkio 3 86 1
memory 7 116 1
devices 2 86 1
freezer 6 21 1
net_cls 5 21 1
I tried adding cgroup_enable=pids
to /boot/cmdline.txt however it does not mount pids cgroup.
In fact, the kernel configuration file shows that this cgroup is not enabled:
$ uname -ar; zgrep "PIDS" /proc/config.gz
Linux kube-master 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux
# CONFIG_CGROUP_PIDS is not set
For now, I see two options:
- Use Kubernetes 1.13.x or lower version, or
- Recompile Raspbian enabling CONFIG_CGROUP_PIDS
@alexellis, any inputs on that?
from k8s-on-raspbian.
I hit this problem as well, can confirm downgrading to 1.13.5 is working fine.
from k8s-on-raspbian.
I have the same Error. Unfortunatly downgrading does not do the trick for me :(
from k8s-on-raspbian.
Is there a good guide/tutorial on how to recompiling Raspbian kernel with CONFIG_CGROUP_PIDS enabled?
from k8s-on-raspbian.
Recompiling Raspbian kernel also worked for me. Here is the guide I followed: Kernel building. Add CONFIG_CGROUP_PIDS=y to "arch/arm/configs/bcmrpi_defconfig" (Raspbian source code).
from k8s-on-raspbian.
should a rpi-update fix this yet? Not sure how to tell when the firmware for rpi-4.19.y will be released.
from k8s-on-raspbian.
I met the same issue on k8s v1.14.1
me too, k8s v1.14.2
from k8s-on-raspbian.
I was able to get my cluster upgraded to 1.15.0
this afternoon by switching the cgroup-driver to systemd.
This does have the issue: Jun 23 20:06:45 rpi-kube-master-01 kubelet[329]: W0623 20:06:45.625225 329 qos_container_manager_linux.go:139] [ContainerManager] Failed to reserve QoS requests: failed to set supported cgroup subsystems for cgroup [kubepods burstable]: Failed to find subsystem mount for required subsystem: pids
however kubelet continues to run.
I don't use any burstable pods on my cluster, but until 4.19.46-v7+
becomes stable it may be a work around.
from k8s-on-raspbian.
Given the issues we are finding with kubeadm
on ARMHF, I am thinking about offering an alternate set of instructions to use k3s as per: https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/
What do you think?
from k8s-on-raspbian.
I'm not sure how k3s would solve the issue at hand? Most of these are issues with either kube-proxy (iptables) and low level cgroups being available from the kernel, which were related to raspbian releases.
I've been running my kubeadm cluster for a few months now, and the only issues I find are during upgrades (OS and kubernetes).
The cluster in it's self is stable for day to day operations and requires little/no picking up.
That being said, I have wondered about moving over to k3s as my etcd instance no longer fits along side the control plane and I have a dedicated RPi for etcd.
My cluster consists of 3x amd64, 2x pine64(arm64), 2x RPi "masters" and 7x RPi workers/slaves.
from k8s-on-raspbian.
My current recommendation is to use k3s, it uses far fewer resources and works on ARM very well, no timing issues.
https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/
Please try it and let us know if it resolves those issues.
from k8s-on-raspbian.
@Mike-Dunton I was able to rpi-update to 4.19.46-v7+ today, and confirm the PIDS fix is in place. kubeadm 1.14.3 installs and inits fine.
$ sudo rpi-update
solved this and made the pids
cgroup available, thank you.
from k8s-on-raspbian.
Thanks for the comments.
I'm closing / archiving this issue now as it seems to have gone off topic.
My recommendation is that all RPi users try k3s which is GA, compliant and better tested for RPi than kubeadm.
https://blog.alexellis.io/raspberry-pi-homelab-with-k3sup/
from k8s-on-raspbian.
PRs are still welcome for the kubeadm guide, just make sure they are tested and have specifics such as versions of Raspbian used and any other steps you ran.
from k8s-on-raspbian.
Related Issues (15)
- question - One node cluster - memory footprint HOT 1
- Readers' RPi clusters HOT 5
- I am getting http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused. error. HOT 11
- Getting a few errors when trying to create my cluster HOT 8
- sudo kubeadm init always fails HOT 2
- Update swap disable instructions in GUIDE.md HOT 5
- "Install Docker" missing slash
- No curl response from pod service HOT 8
- Unattributed cut&paste? HOT 1
- GUIDE.md, "Start up the Kubernetes dashboard" section, outdated? HOT 1
- My Raspberry Pi Kubernetes cluster HOT 1
- Dashboard in PI Cluster HOT 11
- /proc/sys/net/bridge/bridge-nf-call-iptables not found HOT 3
- Flannel installation yaml file link is out of date HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k8s-on-raspbian.