Comments (5)
dperetti
commented on August 26, 2024
2
This is happening to me right now in a DO cluster 🤔.
Update: this was apparently because the node was tainted.
from csi-digitalocean.
ibabiankou
commented on August 26, 2024
1
This seems to be related to kubernetes/kubernetes#68688 Here is bit from kubelet logs:
Oct 11 20:51:49 worker-2 kubelet[15237]: E1011 20:51:49.803324 15237 reflector.go:134] object-"kube-system"/"digitalocean": Failed to list *v1.Secret: secrets "digitalocean" is forbidden: User "system:node:worker-2" cannot list resource "secrets" in API group "" in the namespace "kube-system"
Oct 11 20:52:14 worker-2 kubelet[15237]: E1011 20:52:14.329569 15237 plugin_watcher.go:115] error failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/com.digitalocean.csi.dobs/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration when handling create event: "/var/lib/kubelet/plugins/com.digitalocean.csi.dobs/csi.sock": CREATE
Oct 11 21:07:49 worker-2 kubelet[15237]: I1011 21:07:49.008666 15237 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-ae7c809ecd9911e8" (UniqueName: "kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058") pod "mariadb-server-1" (UID: "ae970759-cd99-11e8-9c23-aacf753f3491")
Oct 11 21:07:49 worker-2 kubelet[15237]: E1011 21:07:49.008912 15237 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058\"" failed. No retries permitted until 2018-10-11 21:07:49.508852241 +0000 UTC m=+1021.187180439 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-ae7c809ecd9911e8\" (UniqueName: \"kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058\") pod \"mariadb-server-1\" (UID: \"ae970759-cd99-11e8-9c23-aacf753f3491\") "
Oct 11 21:07:49 worker-2 kubelet[15237]: I1011 21:07:49.550907 15237 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-ae7c809ecd9911e8" (UniqueName: "kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058") pod "mariadb-server-1" (UID: "ae970759-cd99-11e8-9c23-aacf753f3491")
Oct 11 21:07:49 worker-2 kubelet[15237]: E1011 21:07:49.551572 15237 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058\"" failed. No retries permitted until 2018-10-11 21:07:50.551520535 +0000 UTC m=+1022.229848736 (durationBeforeRetry 1s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-ae7c809ecd9911e8\" (UniqueName: \"kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058\") pod \"mariadb-server-1\" (UID: \"ae970759-cd99-11e8-9c23-aacf753f3491\") "
Oct 11 21:07:50 worker-2 kubelet[15237]: I1011 21:07:50.555877 15237 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-ae7c809ecd9911e8" (UniqueName: "kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058") pod "mariadb-server-1" (UID: "ae970759-cd99-11e8-9c23-aacf753f3491")
Oct 11 21:07:50 worker-2 kubelet[15237]: E1011 21:07:50.556012 15237 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058\"" failed. No retries permitted until 2018-10-11 21:07:52.555964835 +0000 UTC m=+1024.234293043 (durationBeforeRetry 2s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-ae7c809ecd9911e8\" (UniqueName: \"kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058\") pod \"mariadb-server-1\" (UID: \"ae970759-cd99-11e8-9c23-aacf753f3491\") "
Oct 11 21:07:52 worker-2 kubelet[15237]: I1011 21:07:52.565360 15237 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-ae7c809ecd9911e8" (UniqueName: "kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058") pod "mariadb-server-1" (UID: "ae970759-cd99-11e8-9c23-aacf753f3491")
Oct 11 21:07:52 worker-2 kubelet[15237]: E1011 21:07:52.568745 15237 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058\"" failed. No retries permitted until 2018-10-11 21:07:56.5687151 +0000 UTC m=+1028.247043298 (durationBeforeRetry 4s). Error: "Volume not attached according to node status for volume \"pvc-ae7c809ecd9911e8\" (UniqueName: \"kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058\") pod \"mariadb-server-1\" (UID: \"ae970759-cd99-11e8-9c23-aacf753f3491\") "
Oct 11 21:07:56 worker-2 kubelet[15237]: I1011 21:07:56.582947 15237 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-ae7c809ecd9911e8" (UniqueName: "kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058") pod "mariadb-server-1" (UID: "ae970759-cd99-11e8-9c23-aacf753f3491")
Oct 11 21:07:56 worker-2 kubelet[15237]: I1011 21:07:56.588934 15237 operation_generator.go:1205] Controller attach succeeded for volume "pvc-ae7c809ecd9911e8" (UniqueName: "kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058") pod "mariadb-server-1" (UID: "ae970759-cd99-11e8-9c23-aacf753f3491") device path: "csi-4dc069811a39b35b6593833b27419692ae45af019bdb9f2389591890639f3c1e"
Oct 11 21:07:56 worker-2 kubelet[15237]: I1011 21:07:56.684486 15237 operation_generator.go:498] MountVolume.WaitForAttach entering for volume "pvc-ae7c809ecd9911e8" (UniqueName: "kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058") pod "mariadb-server-1" (UID: "ae970759-cd99-11e8-9c23-aacf753f3491") DevicePath "csi-4dc069811a39b35b6593833b27419692ae45af019bdb9f2389591890639f3c1e"
Oct 11 21:07:56 worker-2 kubelet[15237]: I1011 21:07:56.688428 15237 operation_generator.go:507] MountVolume.WaitForAttach succeeded for volume "pvc-ae7c809ecd9911e8" (UniqueName: "kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058") pod "mariadb-server-1" (UID: "ae970759-cd99-11e8-9c23-aacf753f3491") DevicePath "csi-4dc069811a39b35b6593833b27419692ae45af019bdb9f2389591890639f3c1e"
Oct 11 21:07:56 worker-2 kubelet[15237]: E1011 21:07:56.690089 15237 csi_attacher.go:332] kubernetes.io/csi: attacher.MountDevice failed: driver name com.digitalocean.csi.dobs not found in the list of registered CSI drivers
Oct 11 21:07:56 worker-2 kubelet[15237]: E1011 21:07:56.690544 15237 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058\"" failed. No retries permitted until 2018-10-11 21:07:57.190501154 +0000 UTC m=+1028.868829358 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"pvc-ae7c809ecd9911e8\" (UniqueName: \"kubernetes.io/csi/com.digitalocean.csi.dobs^b510b854-cd99-11e8-8aa4-0a58ac148058\") pod \"mariadb-server-1\" (UID: \"ae970759-cd99-11e8-9c23-aacf753f3491\") : driver name com.digitalocean.csi.dobs not found in the list of registered CSI drivers"
from csi-digitalocean.
fatih
commented on August 26, 2024
Thanks for the feedback. Seems like CSI spec is broken in v1.12.0. The way of registration of drivers have changed. We already have NodeGetID() and NodeGetInfo(). I'm looking into now and will post more about it.
from csi-digitalocean.
fatih
commented on August 26, 2024
Here are the steps needed to make the changes for v1.12.0: https://kubernetes-csi.github.io/docs/Setup.html
from csi-digitalocean.
fatih
commented on August 26, 2024
This is now fixed with our latest v0.3.0 release. Here is the changelog for it: https://github.com/digitalocean/csi-digitalocean/blob/master/CHANGELOG.md#v030---20181029
And here is the deployment file: https://github.com/digitalocean/csi-digitalocean/blob/master/deploy/kubernetes/releases/csi-digitalocean-v0.3.0.yaml
from csi-digitalocean.
Related Issues (20)
- Issue creating VolumeSnapshots HOT 3
- Stuck in "Still connecting to unix:///csi/csi.sock" loop
- Problem while existing pod gets updated - multi-attach error. HOT 2
- APIExstension k8s v1beta1 Deprecated
- [Question] privileged vs SYS_ADMIN in the CSI driver deployment
- Deploy as Helm chart HOT 7
- Support for NVMe volumes HOT 10
- [feature request] More forgiveness in rate limiting HOT 9
- PVC - Add do-block-storage-retain storage class with Reclaim policy Retain HOT 4
- Feature request: allow custom snapshot names HOT 1
- Error: waiting for a volume to be created, either by external provisioner "dobs.csi.digitalocean.com" or manually created by system administrator HOT 1
- Round capacity up to minimum supported size HOT 1
- Feature request: Support CSI PV Cloning HOT 5
- Issue with CSI outside of DigitalOcean droplets HOT 1
- Error mounting imported snapshot HOT 5
- Discrepancies with docs about importing snapshots in Kubernetes
- error resizing volume: volume/snapshot capacity limit exceeded HOT 1
- CSI Controller unable to access any API endpoints HOT 1
- Cannot apply csi driver on cis-1.23 hardened rke2 HOT 8
- Feature Request: Support Online PV Resizing
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from csi-digitalocean.