1 slow ops, oldest one blocked for 1880444 sec, mon.e has slow ops about rook HOT 17 OPEN

please find the attached logs

from rook.

Uploading mon-e.log…

from rook.

Uploading operator1.log…

from rook.

attached the operator logs and mon-e logs please let me know how to resolve, its production issue,
we have bloked, due to this we are not able to take backup of it,
we have mounted pvc to this rook-ceph we are not able to go inside pvc from pod, getting permision denied due to this we are not able to take backup

please this is high priority issue

from rook.

Are you saying this issue is preventing you from taking backups?

1 slow ops, oldest one blocked for 1880444 sec, mon.e has slow ops

It is just a health warning, so it is surprising if it is blocking the IO. If it is just mon.e with the problem, you could try scaling down that mon and fail it over to see if a new mon will help.

from rook.

@travisn yes due to this we are not able to take backup because we are using the persistant volumes to take backup, is there any way we can modify the permisions to access the PVC ?

from rook.

@travisn every 45 sec operator will monitor the mon's right it should be resolve by operator, why its not doing it,
is there any thing i have to enable to self heal with operator ?

from rook.

@travisn could you please any advice on how we can enable to self heal with operator
how we can access the mounted pvc backend using ceph storage we are not able to ls as well

from rook.

You can try the direct mount tools

from rook.

@travisn every 45 sec operator will monitor the mon's right it should be resolve by operator, why its not doing it, is there any thing i have to enable to self heal with operator ?

Are you referring to the mon failover? As long as mons are up, this is a different issue.

from rook.

@travisn yes, if mon fails automatically operator pod resolve the issue it should comeup right ?

regards mount issue: if i want to try with directmount tools , how i can confirm that specific workernode, we have issue with sepcific workernode, we are using sharedfilesys as type

from rook.

@travisn yes, if mon fails automatically operator pod resolve the issue it should comeup right ?

Yes, a mon will failover by default after 10 min if it is out of quorum. The doc linked previously explains more.

regards mount issue: if i want to try with directmount tools , how i can confirm that specific workernode, we have issue with sepcific workernode, we are using sharedfilesys as type

If you want to mount it from a specific worker node, you could create the direct tools pod with a node selector.

from rook.

@travisn we can see below logs in the rbdplugin-provisioner, is this causing not able to access the mountpoint getting permision denied

`sudo kubectl logs csi-rbdplugin-provisioner-84c7fb8d76-q68lv -n rook-ceph csi-provisioner
I0330 05:46:23.300526 1 csi-provisioner.go:138] Version: v3.0.0
I0330 05:46:23.300616 1 csi-provisioner.go:161] Building kube configs for running in cluster...
I0330 05:46:26.237022 1 common.go:111] Probing CSI driver for readiness
I0330 05:46:26.262363 1 csi-provisioner.go:277] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I0330 05:46:26.264594 1 leaderelection.go:248] attempting to acquire leader lease rook-ceph/rook-ceph-rbd-csi-ceph-com...
E0402 11:59:38.713354 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": http2: client connection lost
E0402 11:59:48.560986 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 11:59:57.772664 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:00:08.941466 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:00:21.230099 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:00:33.516702 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:00:46.924557 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:00:59.213800 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:01:19.596629 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:01:56.022719 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: i/o timeout
E0402 12:02:03.724542 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:02:11.820950 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:02:22.163660 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:02:34.444823 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:02:45.613050 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:02:55.950080 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:03:05.165761 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:03:14.380690 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:03:23.596301 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:03:34.768516 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:03:48.172259 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:04:00.460616 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:04:09.677223 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:04:17.773506 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:04:26.988783 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:04:33.133365 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:04:40.400824 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:04:49.616259 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:04:58.829464 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:05:08.044470 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:05:20.332322 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:05:31.505712 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:05:40.721216 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:05:49.932707 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:05:59.148667 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:06:09.484996 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:06:17.580295 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:06:23.724642 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:06:30.988966 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:06:43.276684 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:06:51.375085 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:06:58.636941 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:07:09.804808 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:07:19.021395 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:07:29.360543 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:07:38.572435 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:07:47.788678 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:07:53.933206 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:08:03.149706 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:08:12.364779 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:08:23.532832 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:08:32.748570 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:08:52.396059 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:09:00.400787 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:09:07.756605 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:09:18.829735 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:09:24.972737 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:09:37.261601 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:09:47.692587 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:09:53.837106 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:10:03.052785 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:10:14.124800 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:10:26.412531 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:10:35.629243 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:10:46.060566 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:10:54.060405 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:11:00.208714 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:11:07.565208 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:11:18.637217 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:11:29.069262 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:11:35.212642 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:11:47.500755 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:11:55.500846 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:12:07.788130 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:12:17.004366 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:12:27.437062 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:12:36.652573 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:12:44.656821 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:12:52.013114 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:13:01.232566 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:13:07.372911 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:13:15.372713 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:13:22.733054 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:13:31.948449 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:13:39.953512 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:13:50.381272 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:13:58.381209 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:14:05.744735 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:14:18.028188 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:14:27.244434 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:14:36.461118 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:14:42.925195 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:14:50.892791 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:15:00.108824 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:15:13.644301 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:15:22.864233 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:15:33.904216 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:15:46.189274 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:15:56.652687 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:16:04.620458 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:16:21.228767 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:16:33.516896 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:16:41.488778 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:16:51.951549 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:17:01.169980 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:17:09.132360 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:17:21.420462 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:17:28.813211 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:17:39.853202 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:17:49.068529 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:18:01.356403 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:18:11.820531 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:18:21.036272 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:18:27.181333 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:18:35.149229 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:18:45.613478 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:18:53.580896 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:19:04.046448 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:19:13.264448 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:19:22.476297 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:19:31.692120 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:19:39.664726 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:19:45.804853 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:19:51.953883 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:19:58.092973 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:20:08.556717 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:20:14.701560 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:20:26.988600 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:20:38.028577 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:20:48.492900 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:20:59.536662 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:21:13.068234 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:21:21.040706 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:21:33.324472 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:21:40.716653 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:21:49.932762 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:21:57.900258 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:22:05.293232 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:22:14.508730 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:22:22.480354 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:22:29.868722 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:22:40.908943 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:22:50.124657 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:23:02.413056 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:23:09.805301 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:23:17.932702 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:23:28.236275 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:23:36.368000 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:23:46.668746 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:23:57.868865 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:24:07.085172 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:24:17.388420 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:24:26.608410 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:24:32.749756 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:24:40.876970 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:24:50.097049 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host
E0402 12:25:00.481411 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:25:08.456317 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:25:18.669614 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:25:28.259832 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:25:34.996712 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:25:42.593671 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:25:49.077372 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:25:56.942969 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:26:07.221009 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:26:17.822336 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:26:22.960686 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:26:33.745738 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:26:42.524636 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:26:53.055473 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:26:59.788320 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:27:06.213512 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:27:16.988802 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:27:23.354661 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:27:32.476279 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:27:38.472413 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:27:45.142355 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:27:51.698397 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:28:01.039511 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:28:06.322808 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:28:16.884096 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:28:22.012813 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:28:27.383993 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:28:36.051936 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:28:42.110972 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:28:52.030314 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:29:01.565392 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:29:10.887716 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:29:18.493427 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:29:28.473617 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:29:37.330322 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:29:46.437647 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused
E0402 12:29:56.948023 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused

`

from rook.

@travisn rbdplugin-provisioner is mounting emptyDir, what it write in the emptyDir ?

from rook.

Can you provide the following?

• What is the latest output of ceph status? to show if ceph is healthy
• Have you checked the CSI troubleshooting guide?
• The CSI provisioner is only used for new volumes. If you are having problems with existing volumes, it is not related to the provisioner.
• What exactly is the issue? IO from cephfs or rbd volumes is not working? Please summarize the issue, and perhaps @Rakshith-R can take a look if related to CSI.

from rook.

I'm hitting a similar issue on a freshly setup cluster. Please LMK if I should open a new bug instead.

Hardware:
2x10Gbps bonded network connections
3x4TB NVMes per node (first NVMe has a 200GB root partition, rest is dedicated to ceph. Remaining two NVMes are fully dedicated to OSDs)
96 GB RAM per node
11 nodes

Servers are running NixOS 23.11 (Linux oecs3 6.1.91 #1-NixOS SMP PREEMPT_DYNAMIC Fri May 17 09:56:25 UTC 2024 x86_64 GNU/Linux)

Our rook configuration doesn't have anything complicated. 3 OSDs/node and 3 mon daemons for the cluster, constrained to run on the first 3 servers.

Symptoms I see:

• mons failing healthchecks (mostly because of ceph commands timing out)
• mons eventually get replaced (at the moment we're running mon.{d,e,f} because a,b, and c were destroyed.
• slow mons seem to use 100% of a single core. All mons get into this state eventually (and get replaced).
• very often I see "slow_ops" messages in the logs, coming from the mons.
• cluster is constantly in an unhealthy state - never recovering.

I just tried debugging by attaching strace to the mon process using 100% CPU on a single core. strace log is filled with close() calls (millions of them within a minute or so), with no open or other calls. Sample below:

close(640694286)                        = -1 EBADF (Bad file descriptor)
close(640694287)                        = -1 EBADF (Bad file descriptor)
close(640694288)                        = -1 EBADF (Bad file descriptor)
close(640694289)                        = -1 EBADF (Bad file descriptor)
close(640694290)                        = -1 EBADF (Bad file descriptor)
close(640694291)                        = -1 EBADF (Bad file descriptor)
close(640694292)                        = -1 EBADF (Bad file descriptor)
close(640694293)                        = -1 EBADF (Bad file descriptor)
close(640694294)                        = -1 EBADF (Bad file descriptor)
close(640694295)                        = -1 EBADF (Bad file descriptor)
close(640694296)                        = -1 EBADF (Bad file descriptor)

Note that this is a fresh cluster with no workload.

Hope this helps

EDIT: attached logs from the mon
mon-e.log

from rook.

I noticed similar issues reported. As pointed out in this comment the issue seems to stem from LimitNOFILE=infinity.

More specifically, for our setup, NixOS 23.11 (or -unstable) hasn't gotten the patch from upstream containerd (see: https://github.com/NixOS/nixpkgs/blob/a8695cbd09a7ecf3376bd62c798b9864d20f86ee/nixos/modules/virtualisation/containerd.nix#L87).

For people using NixOS, this seems to work:

  virtualisation.containerd.enable = true;
  systemd.services.containerd.serviceConfig = {
    LimitNOFILE = lib.mkForce null;
  };

from rook.