Comments (2)
Thanks @neoaggelos - the --all-containers
log flag identified the problem source. The rbd plugin is unable to load the rbd
kernel module:
chrrles@misscompy:~$ kubectl logs csi-rbdplugin-5nvp9 -n rook-ceph --all-containers
...snip...
E0112 01:51:42.259532 42440 rbd_util.go:303] modprobe failed (an error (exit status 1) occurred while running modprobe args: [rbd]): "modprobe: ERROR: could not insert 'rbd': Exec format error\n"
Manually loading the rbd
module and reloading then results in a warning that the nbd
module cannot be loaded:
chrrles@misscompy:~$ kubectl delete pods csi-rbdplugin-5nvp9 -n rook-ceph
pod "csi-rbdplugin-5nvp9" deleted
chrrles@misscompy:~$ kubectl logs ds/csi-rbdplugin -n rook-ceph --all-containers
W0112 01:57:48.892866 59135 rbd_attach.go:226] nbd modprobe failed (an error (exit status 1) occurred while running modprobe args: [nbd]): "modprobe: ERROR: could not insert 'nbd': Exec format error\n"
chrrles@misscompy:~$ sudo modprobe nbd
chrrles@misscompy:~$ kubectl delete pods/csi-rbdplugin-5n5n9 -n rook-ceph
pod "csi-rbdplugin-5n5n9" deleted
chrrles@misscompy:~$ kubectl logs ds/csi-rbdplugin -n rook-ceph --all-containers
I0112 01:59:58.670031 65930 main.go:167] Version: v2.7.0
I0112 01:59:58.670086 65930 main.go:168] Running node-driver-registrar in mode=registration
I0112 01:59:59.677013 65930 node_register.go:53] Starting Registration Server at: /registration/rook-ceph.rbd.csi.ceph.com-reg.sock
I0112 01:59:59.677312 65930 node_register.go:62] Registration Server started at: /registration/rook-ceph.rbd.csi.ceph.com-reg.sock
I0112 01:59:59.677437 65930 node_register.go:92] Skipping HTTP server because endpoint is set to: ""
I0112 02:00:00.572303 65930 main.go:102] Received GetInfo call: &InfoRequest{}
I0112 02:00:00.573246 65930 main.go:109] "Kubelet registration probe created" path="/var/snap/microk8s/common/var/lib/kubelet/plugins/rook-ceph.rbd.csi.ceph.com/registration"
I0112 02:00:00.633474 65930 main.go:121] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
Adding rbd
and nbd
to /etc/modules fixes the issue with mounting the PVC volume. Further research shows this to be a kernel incompatibility preventing the rbd module from being loaded by the rook agent... Which is odd because this kernel should be compatible (https://documentation.suse.com/es-es/ses/7/html/ses-all/admin-caasp-ceph-common-issues.html#solution-7):
chrrles@misscompy:~/ollama$ uname -a
Linux misscompy 6.5.0-14-generic #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:59:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Regardless, this does not feel related to microk8s so I will close this issue. Thanks again for helping @neoaggelos !
💯 🎆 🙌
from microk8s.
Hi @chrrrles
Looks like Rook is having trouble registering the CSI drivers and nodes? Can you check the output of:
sudo microk8s kubectl get csidrivers
sudo microk8s kubectl get csinodes
Also, retrieving logs from --all-containers
on the rook pods might shed some more light as to what is happening. Can you also check whether the ceph rbd pools are created? Though I imagine the issues are related to CSI instead.
from microk8s.
Related Issues (20)
- Problem running a container witth mounted empty dir with memory medium
- MicroK8s 1.28.3 node not ready
- Can't get nvidia-smi to work in a pod HOT 1
- Is it Possible to Change the Expiry Dates in DLL files HOT 2
- CVE-2024-21626 Vulnerability: Attac/Access to HOST Filesystem via "runc" Leak for versions 1.26 - 1.29 HOT 3
- microk8s connect-external-ceph
- DNS resolving only works for the node coredns pod is on HOT 2
- re-enabling ha-cluster renders the cluster unusable
- nfs fails to mount with "bad option"
- Raspberry Pi pods stuck in: failed to create shim task: OCI runtime create failed ... unified resource "memory.oom.group" can't be set HOT 1
- microk8s is not running. Use microk8s inspect for a deeper inspection.
- Creating two containers of WSO2 Apim (Active-Active Setup) HOT 1
- Snap install fails on long hostnames HOT 1
- Enable cilium addon fails in multinode cluster HOT 1
- microk8s is not running - on a 4 node Rasp Pi 3 B+ cluster HOT 2
- Microk8s can't get swap to enable HOT 2
- Fresh install of microK8s won't come up HOT 3
- Can't apply some K8s manifests because of undefined symbol and GLIBC_PRIVATE HOT 1
- Cannot enable gpu addon on aarch64 instances (aws g5g.metal) HOT 3
- Microk8s stuck and cannot connect to VM on MacOS HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from microk8s.