Deployment by README steps fails in 4-node cluster about csi-driver-host-path HOT 13 CLOSED

from csi-driver-host-path.

@pohly proposed these changes to get example deployment working:

• start one instance of csi-hostpathplugin
• use inter-pod affinity to land attacher, provisioner, hostpathplugin and my-csi-app on same node

If we agree on this, I will prepare a PR

from csi-driver-host-path.

I observe the same error:

Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  FailedScheduling        17s (x4 over 17s)  default-scheduler        pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
  Normal   Scheduled               17s                default-scheduler        Successfully assigned gitops/atlantis-0 to 10.0.1.78
  Normal   SuccessfulAttachVolume  17s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-7b2a9f04-36ae-11e9-8852-52540082fa9b"
  Warning  FailedMount             1s (x2 over 1s)    kubelet, 10.0.1.78       MountVolume.SetUp failed for volume "pvc-7b2a9f04-36ae-11e9-8852-52540082fa9b" : rpc error: code = Unknown desc = mount failed: exit status 255
Mounting command: mount
Mounting arguments: -o bind /tmp/7b4198ab-36ae-11e9-8baf-5254003ed451 /opt/rke/var/lib/kubelet/pods/7b2de1a5-36ae-11e9-8852-52540082fa9b/volumes/kubernetes.io~csi/pvc-7b2a9f04-36ae-11e9-8852-52540082fa9b/mount
Output: mount: mounting /tmp/7b4198ab-36ae-11e9-8baf-5254003ed451 on /opt/rke/var/lib/kubelet/pods/7b2de1a5-36ae-11e9-8852-52540082fa9b/volumes/kubernetes.io~csi/pvc-7b2a9f04-36ae-11e9-8852-52540082fa9b/mount failed: No such file or directory

On this host 10.0.1.78, however, see:

[rancher@rancher ~]$ sudo ls -la /opt/rke/var/lib/kubelet/pods/7b2de1a5-36ae-11e9-8852-52540082fa9b/volumes/kubernetes.io~csi/pvc-7b2a9f04-36ae-11e9-8852-52540082fa9b
total 12
drwxr-x---    2 root     root          4096 Feb 22 14:47 .
drwxr-x---    3 root     root          4096 Feb 22 14:47 ..
-rw-r--r--    1 root     root           248 Feb 22 14:48 vol_data.json

Therefore, only the .../mount folder is missing. It's seems this comes from the request
however this path is at least not created by this csi-driver-driver-hostpath

from csi-driver-host-path.

In other words: How would a proper multi node example look like?

from csi-driver-host-path.

Hostpath by design can only work on a single node because it allocates volumes locally an a node. The solution I mentioned to @okartau ensures that a single hostpath driver runs and that everything else (provisioner, attacher, and the app) run on that same node.

This is not a good example because it forces a special deployment of the app. The advantage is that it works without depending on alpha features.

A better example would be to run provisioner, attacher and driver on one node and then force the app onto that node via topology. But that depends on alpha features in Kubernetes and is more complicated, therefore I think we should do the easy solution first and add topology support later.

from csi-driver-host-path.

@pohly Thanks for you comment.

I understand that the volume is bound to a host, and can only reasonably be consumed by apps deployed as a StatefulSet. However, I do not yet understand, why the provisioner and attacher cannot run successfully on multiple hosts, for example to cater a default StorageClass with provider: csi-hostpath.

Is that a limitation of the CSI spec? - I guess I'm still missing some bits, here.

from csi-driver-host-path.

Provisioner and attacher can only talk to a CSI driver that runs on the same host. That's because communication happens via a local Unix domain socket.

The usual model for other drivers that talk to some external storage backend is to run the CSI driver on all nodes and the attacher and provisioner on a few nodes (replicated, with leader election to determine which instance actually responds to requests).

Or at least that is the desirable model. In practice some deployments only start provisioner and attacher once, which causes delays during provisioning/attaching when those pods need to be restarted.

from csi-driver-host-path.

Thank you that clarifies a lot. In theory, host-path plugin - based on the request metadata - could determine on which node operations of provision / attach should ultimately take place.

That's unless, conceptually, it would be responsibility of a custom provisioner / attacher, which is actually what their substantivated names would indicate in my understanding.

Assuming I understood things the way they are: is supporting the mentioned* use case considerable for this project or completely out of scope?

* Create multi node cluster default StorageClass with provider: csi-hostpath to cater to PVCs

from csi-driver-host-path.

I think, this is mainly caused by controllerserver which connets to external-provisoner by local socket and only support create volume locally in csi-driver-host-path. However, nodeserver runs as a daemonset which may not run on the same node as external-provisoner.

from csi-driver-host-path.

@cwdsuzhou daemonset ensures that there is a hostpath driver instance on all nodes, so external-provisioner always has a local driver to talk to. The problem is that the app then also needs to run on that same node, which currently isn't enforced.

@okartau has a fix in #14

from csi-driver-host-path.

@cwdsuzhou daemonset ensures that there is a hostpath driver instance on all nodes, so external-provisioner always has a local driver to talk to. The problem is that the app then also needs to run on that same node, which currently isn't enforced.

@okartau has a fix in #14

Thank you! I know making sure the apps run the same node can fix it. What I want to describe is that the nodeserver is actually working for the apps should run on the same node with the external-provsioner.

from csi-driver-host-path.

@cwdsuzhou what do you mean with "nodeserver is actually working for the apps"?

from csi-driver-host-path.

@cwdsuzhou what do you mean with "nodeserver is actually working for the apps"?
It means the apps should run the same node with the external-provisoner and attacher

from csi-driver-host-path.