Comments (7)
codenrhoden
commented on June 12, 2024
8
Okay, one more thing for you to check...
Can you see if disk.EnableUUID is set for the VM that you are using as a K8s worker node? When reminding myself about this parameter I went looking through our docs, and realized it's not called out anywhere specifically in the CSI docs, or the out-of-tree cloud provider, only the older in-tree vSphere cloud provider, here. That's not good on our part, as this parameter is required to be set on all K8s worker VMs.
@dvonthenen @frapposelli @andrewsykim I'm not sure if we need to document this requirement at the CSI level, the CCM level, or both maybe?
from vsphere-csi-driver.
codenrhoden
commented on June 12, 2024
Hi @qw1mb0,
Thanks for attaching the detailed logs. That helps.
The way the driver would find that the disk is attached is by looking in /dev/disk/by-id. Specifically in your case, it would be looking for the disk at
/dev/disk/by-id/wwn-0x6000c2917fad296049d3923d725e773e. If you are still able to look at this exact setup, can you see if the disk is attached there?
from vsphere-csi-driver.
qw1mb0
commented on June 12, 2024
I capped the problem cluster and put the cluster kubernetes 1.15.
Install vsphere-csi-plugin again.
Created a test under PVC:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-vsphere-csi-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: vsphere-main
---
kind: Pod
apiVersion: v1
metadata:
name: my-csi-app
spec:
containers:
- name: my-frontend
image: busybox
volumeMounts:
- mountPath: "/data"
name: my-fcd-volume
command: [ "sleep", "1000000" ]
volumes:
- name: my-fcd-volume
persistentVolumeClaim:
claimName: my-vsphere-csi-pvc
There is this problem in the bottom of the deck:
Warning FailedMount 3s (x4 over 7s) kubelet, csi-master MountVolume.MountDevice failed for volume "pvc-4fb15e07-a8a1-4eb9-a193-488481eca7ee" : rpc error: code = NotFound desc = disk: 6000c2911aa586adcd933ddd12c97c10 not attached to node
The node has this disk:
# lsblk | grep sdd
sdd 8:48 0 5G 0 disk
In controller logs:
E0711 18:58:10.509672 1 datacenter.go:535] GetFirstClassDiskByName failed. Err: No vSphere disk ID/Name found
I0711 18:58:11.702982 1 datacenter.go:610] DoesFirstClassDiskExist(ec2f91bf-b652-4022-891a-e66c02f89e42): FOUND
time="2019-07-11T18:58:12Z" level=info msg="AttachDisk([3par_4_Lun101] fcd/d978945e76fb43878da40d0e4d1684df.vmdk) succeeded with: VolID=ec2f91bf-b652-4022-891a-e66c02f89e42 UUID=6000c2911aa586adcd933ddd12c97c10"
In vsphere-csi-node logs:
time="2019-07-11T18:58:54Z" level=debug msg="/csi.v1.Node/NodeGetCapabilities: REQ 0018: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-07-11T18:58:54Z" level=debug msg="/csi.v1.Node/NodeGetCapabilities: REP 0018: Capabilities=[rpc:<type:STAGE_UNSTAGE_VOLUME > ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-07-11T18:58:54Z" level=debug msg="/csi.v1.Node/NodeStageVolume: REQ 0019: VolumeId=8115dfb9-1408-494e-b0a6-2292f4fd03ec, PublishContext=map[datacenter:X2 name:pvc-a3eadaed-a8cd-4383-ac0b-74cda234e0ca page83data:6000c29cab1f47b6adb47f0bc816d4c1 parent_name:LUN101 parent_type:Datastore type:First Class Disk vcenter:vcenter.example.com], StagingTargetPath=/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a3eadaed-a8cd-4383-ac0b-74cda234e0ca/globalmount, VolumeCapability=mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > , VolumeContext=map[datacenter:X2 name:pvc-a3eadaed-a8cd-4383-ac0b-74cda234e0ca parent_name:LUN101 parent_type:Datastore storage.kubernetes.io/csiProvisionerIdentity:1562871196097-8081-vsphere.csi.vmware.com type:First Class Disk vcenter:vcenter.exampl.ecom], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-07-11T18:58:54Z" level=debug msg="checking if volume is attached" diskID=6000c29cab1f47b6adb47f0bc816d4c1 volID=8115dfb9-1408-494e-b0a6-2292f4fd03ec
time="2019-07-11T18:58:54Z" level=debug msg="/csi.v1.Node/NodeStageVolume: REP 0019: rpc error: code = NotFound desc = disk: 6000c29cab1f47b6adb47f0bc816d4c1 not attached to node"
There is no such disk in /dev/disk/by-id/:
# ls -lah /dev/disk/by-id/
total 0
drwxr-xr-x 2 root root 120 Jul 11 21:33 .
drwxr-xr-x 6 root root 120 Jul 11 21:33 ..
lrwxrwxrwx 1 root root 9 Jul 11 21:33 ata-VMware_Virtual_SATA_CDRW_Drive_00000000000000000001 -> ../../sr0
lrwxrwxrwx 1 root root 10 Jul 11 21:33 dm-name-ubuntu--image--new-root -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jul 11 21:33 dm-uuid-LVM-BHiKm4tfbEz8weWiguUUoVbbwssmVxEQ2CyU08eoBSKxBcmBEXSvi3kA3NzUqmzn -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jul 11 21:33 lvm-pv-uuid-yYxQdE-vJxW-xor5-Q5ef-QDCg-yFX1-K2LJpS -> ../../sda3
[lm-api-manager] root@csi-master ~ # ls -lR /dev/disk/
/dev/disk/:
total 0
drwxr-xr-x 2 root root 120 Jul 11 22:06 by-id
drwxr-xr-x 2 root root 100 Jul 11 22:06 by-partuuid
drwxr-xr-x 2 root root 200 Jul 11 22:06 by-path
drwxr-xr-x 2 root root 100 Jul 11 22:06 by-uuid
/dev/disk/by-id:
total 0
lrwxrwxrwx 1 root root 9 Jul 11 22:06 ata-VMware_Virtual_SATA_CDRW_Drive_00000000000000000001 -> ../../sr0
lrwxrwxrwx 1 root root 10 Jul 11 22:06 dm-name-ubuntu--image--new-root -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jul 11 22:06 dm-uuid-LVM-BHiKm4tfbEz8weWiguUUoVbbwssmVxEQ2CyU08eoBSKxBcmBEXSvi3kA3NzUqmzn -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jul 11 22:06 lvm-pv-uuid-yYxQdE-vJxW-xor5-Q5ef-QDCg-yFX1-K2LJpS -> ../../sda3
/dev/disk/by-partuuid:
total 0
lrwxrwxrwx 1 root root 10 Jul 11 22:06 db034a7a-cca8-4558-9e75-133124a9fe9d -> ../../sda3
lrwxrwxrwx 1 root root 10 Jul 11 22:06 ee185204-9faa-42e0-80f3-c92390225f7b -> ../../sda2
lrwxrwxrwx 1 root root 10 Jul 11 22:06 fe9123c3-733b-438b-bcfa-534f9cf96c96 -> ../../sda1
/dev/disk/by-path:
total 0
lrwxrwxrwx 1 root root 9 Jul 11 22:06 pci-0000:02:01.0-ata-1 -> ../../sr0
lrwxrwxrwx 1 root root 9 Jul 11 22:06 pci-0000:03:00.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Jul 11 22:06 pci-0000:03:00.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jul 11 22:06 pci-0000:03:00.0-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jul 11 22:06 pci-0000:03:00.0-scsi-0:0:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Jul 11 22:06 pci-0000:03:00.0-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx 1 root root 9 Jul 11 22:06 pci-0000:03:00.0-scsi-0:0:2:0 -> ../../sdc
lrwxrwxrwx 1 root root 9 Jul 11 22:06 pci-0000:03:00.0-scsi-0:0:3:0 -> ../../sdd
/dev/disk/by-uuid:
total 0
lrwxrwxrwx 1 root root 10 Jul 11 22:06 19444aac-c1da-4d59-8cc1-5513f2d2329a -> ../../sda2
lrwxrwxrwx 1 root root 9 Jul 11 22:06 34fccb7d-3d28-465c-ba3d-baffe1249288 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jul 11 22:06 ede9542d-0701-4da0-be85-38a0ed6bbb76 -> ../../dm-0
from vsphere-csi-driver.
qw1mb0
commented on June 12, 2024
facepalm...
You are right, we forgot about disk.EnableUUID.
This is in the documentation: https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/master/docs/deploying_csi_vsphere_with_rbac.md#vsphere
from vsphere-csi-driver.
codenrhoden
commented on June 12, 2024
This is in the documentation: https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/master/docs/deploying_csi_vsphere_with_rbac.md#vsphere
Well darn, I was git grep-ing through our docs for enableUUID or enableuuid, but didn't capitalize the E, since the original docs didn't write it that way. Glad you found it!
from vsphere-csi-driver.
qw1mb0
commented on June 12, 2024
thank you very much for the quick response
from vsphere-csi-driver.
EldinEgrlic
commented on June 12, 2024
Okay, one more thing for you to check...
Can you see if
disk.EnableUUIDis set for the VM that you are using as a K8s worker node? When reminding myself about this parameter I went looking through our docs, and realized it's not called out anywhere specifically in the CSI docs, or the out-of-tree cloud provider, only the older in-tree vSphere cloud provider, here. That's not good on our part, as this parameter is required to be set on all K8s worker VMs.@dvonthenen @frapposelli @andrewsykim I'm not sure if we need to document this requirement at the CSI level, the CCM level, or both maybe?
@codenrhoden
Thanks, this helped me.
from vsphere-csi-driver.
Related Issues (20)
- Block volume sometimes takes 5 minutes HOT 4
- Format options HOT 2
- Remove dependency on control plane nodes HOT 4
- existing volume not found by cns after vcenter migration HOT 2
- specify kube-apiserver address
- Unable to Resize FS after Online PVC Expansion HOT 3
- QueryVolumes not found after cluster teardown / recreation HOT 1
- Crashing vsphere-csi-controller with RWX (ReadWtiteMany) PV HOT 10
- Update sidecar containers to fix CVEs and others
- Provisioner, attacher,... too long to respond after a crash HOT 3
- Can't extend volume "CnsFault error: VSLM task failed" HOT 8
- Rebuild base images at release time to fix CVEs HOT 6
- Impossible to exceed the limit of 59 pvc per worker HOT 4
- vSphere CSI Node Pods Consistently CrashLoopBackOff HOT 1
- Issue with upgrading vsphere-csi-controller from 3.0.3 to 3.1.x in a topology aware setup. HOT 6
- Can't read superblock on /dev/sdi HOT 2
- thumbprint based authentication does not work when thumbprint is specified in the VirtualCenter section in vSphere Config Secret
- Cross NS quotas
- volumeID not found in QueryVolume HOT 1
- Volume File Share RWX Expand
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vsphere-csi-driver.