Pod with PVC provisioned by csi-driver-nfs fails to start due to FailedMount with error `mkdir /var/snap: read-only file system` about csi-driver-nfs HOT 7 CLOSED

it seems the nfs csi driver does not have permission to mkdir /var/snap

from csi-driver-nfs.

I have pretty same setup and facing the same problem

• MicroK8s v1.27.5 revision 5891
• TrueNAS-13.0-U5.3
• CSI Driver version: helm.sh/chart: csi-driver-nfs-v4.4.0

Error message for the pod being created

MountVolume.SetUp failed for volume "pvc-dd295906-c30a-489a-b129-2779783d6a26" : rpc error: code = Internal desc = mkdir /var/snap: read-only file system

Can confirm that other NFS-based persistence implementations, like https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/ do not suffer from this problem, i am able to create PVC+PV with this solution and it mounted on the node under
/var/snap/microk8s/common/var/lib/kubelet/pods/1f78a43a-6fef-4353-b51b-3aeb66e999c8/volumes/kubernetes.io~nfs/pvc-3901c382-754c-427d-b972-c415c6f98afc

from csi-driver-nfs.

it seems the nfs csi driver does not have permission to mkdir /var/snap

Yeah, though that's the error quoted from the digging that I've done that error doesn't make any sense. The pod isn't mounting any internal volumes under /var/snap, so it should/will have permission to that path (if it even exists within the pod). That would mean it's the driver itself trying to mkdir /var/snap on the host machine, but that probably shouldn't happen either. I can confirm that the nfs csi driver is able to make directories on the attached storage, as it makes the root directory following the naming scheme described in the storage class.

if the nfs csi driver is attempting to make any directories under /var/snap and not under directories that belong to microk8s itself or something similar, there'll be obvious permission problems because Snap locks down the filesystem to prevent abuse. I did try opening up the permissions in /var/snap just to see if that changed anything, but it caused my entire cluster to fail.

from csi-driver-nfs.

I was able to install version 4.2.0 and I am now able to mount PVCs to my NAS. I have found file permission issues that seem to appear to stem from the csi-driver-nfs itself, as I can use the same mount command via CLI and have no issues. So the original issue reported appears to be with newer versions of the csi driver

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
csi-driver-nfs  kube-system     5               2023-10-23 04:41:03.869148025 +0000 UTC deployed        csi-driver-nfs-v4.2.0   v4.2.0    

from csi-driver-nfs.

I also have the same error...

• microk8s 1.24.17
• csi-provisioner:v3.5.0
• livenessprobe:v2.10.0
• snapshot-controller:v6.2.2

My error is...

MountVolume.SetUp failed for volume "pvc-68e70909-9c25-44bc-933a-c1ddffcf02d9" : rpc error: code = Internal desc = mkdir /var/snap: read-only file system

from csi-driver-nfs.

I was able to fix the immediate error mkdir /var/snap: read-only file system by downgrading to v4.2.0, however I still had issues actually writing not to the NFS but to the intermediate file at /tmp/pvc-*. This was because the official documentation suggests to use the flag --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet", however per the microk8s docs we should actually use --set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet, exluding the linux. prefix. This fixed all issues for me, and it may fix the mkdir /var/snap: read-only file system error in later versions.

from csi-driver-nfs.

--set kubeletDir is the right setting, I have worked out a doc PR to fix this issue: #537

from csi-driver-nfs.