ccremer / kubernetes-zfs-provisioner Goto Github PK

Example: https://github.com/rimusz/charts/blob/master/stable/hostpath-provisioner/templates/storageclass.yaml#L11-L14

Docs: https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

I can open a PR

Thanks for this awesome software! It fits perfectly in a niche position in my project.
As I can see currently update-permissions.sh runs chmod g+w to newly created datasets. However, these dataset mountpoints are still owned by root:root, thus an ordinary user in k8s pods will not be able to write there. Can you explain what the point is about g+w? Can we change it to o+w or do a chown to a specific user instead?

Hi I would like to commit my umbrella helm chart of kubernetes-zfs-provisioner to git so ArgoCD can manage it. To do this I of course cannot commit my ssh key since that is a big no-no. Is there a way I can omit this from values.yaml by either using secrets or something else to load the key?

Hi! You have a pretty neat project here.

Have you considered using zfs send/recv to migrate datasets between nodes to work around datasets being locked to a particular node?

I'm imagining a few potential designs:

1. When a dataset is on one node and the pod that needs it is scheduled on a different node, zfs send it to the new node and drop it from the current node.
2. A main backup/restore pool that maintains the canonical copies of datasets
   • When a pod is scheduled on a node, the node acquires a lease on any datasets it needs and recv's the data from the main pool.
   • The pod can start once the dataset is received.
   • Data is saved directly to the node, and regular snapshots are streamed from the node back to the main pool.
     • If the node dies or is network partitioned, you can manually break the lease, losing any data since the last snapshot. With frequent snapshots this is still a pretty good outcome considering a node with the only copy of the leased dataset just vanished.
   • When the pod is unscheduled and the containers using the dataset stop, it takes a final snapshot, sends it to the main pool and releases the lease.

Thoughts?

Would it be possible to add ARM64 images to your GitHub Action pipeline? I have an example here for reference.

As-is, the pods immediately crash:

standard_init_linux.go:219: exec user process caused: exec format error

As in issue #85, my ZFS host is part of the cluster as a worker node. I'm not a real fan of using SSH from the provisioner container to run commands on the ZFS host and I'd much prefer running the provisioner as a daemon directly on the ZFS host.

So far I've been able to make it work by running kubernetes-zfs-provisioner directly on the ZFS host with the ZFS_KUBE_CONFIG_PATH environment variable pointing to my "admin" kubeconfig.

Obviously this is not ideal because the "admin" user permissions are too open for what the provisioner has to do. What would be the right thing to do instead here?

It also required this tiny change:

diff --git a/pkg/zfs/zfs.go b/pkg/zfs/zfs.go
index 015fa16d78db..cc3196bc3269 100644
--- a/pkg/zfs/zfs.go
+++ b/pkg/zfs/zfs.go
@@ -114,7 +114,7 @@ func (z *zfsImpl) SetPermissions(dataset *Dataset) error {
 	if dataset.Mountpoint == "" {
 		return fmt.Errorf("undefined mountpoint for dataset: %s", dataset.Name)
 	}
-	cmd := exec.Command("update-permissions", dataset.Hostname, dataset.Mountpoint)
+	cmd := exec.Command("chmod", "g+w", dataset.Mountpoint)
 	out, err := cmd.CombinedOutput()
 	if err != nil {
 		return fmt.Errorf("could not update permissions on '%s': %w: %s", dataset.Hostname, err, out)

The change is quite simple but it does break the "normal" use-case though. I'm not sure how we could make it more generic.

Deployment fails with error message in title.

I believe this is an upstream issue: kubernetes-sigs/sig-storage-lib-external-provisioner#123. As mentioned in the linked issue, the problem was actually fixed in upstream but a release with that fix was never made. This project is using lateset release, v8, of storage-lib-external-provisioner which has the problem.

I took a look at the nfs-subdir-external-provisioner to see how they managed to mitigate this issue, and turns out they limited the version of storage-lib-external-provisioner to v6, which doesn't have the problematic line of code. I think we can probably do the same before the upstream release the fix to at lease make this project usable?

Currently space is reserved when the dataset is created. What do you think about making this optional, essentially making the PVs thinly-provisioned? This should probably be an option in the storage class.

It seems that this provisioner currently does not support resizing existing PVC's. Would you consider adding support for that?

Error: UPGRADE FAILED: cannot patch "test-service" with kind PersistentVolumeClaim: persistentvolumeclaims "test-service" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

This has been bugging me.

I'm currently using hostpath-provisioner and have /nfs/hostpath on all nodes. It's a nfs mount on all nodes except the zfs node and works elegantly. But I would rather the provisioner did zfs create for every pv/pvc.

Not sure if performance is worse when using nfs on localhost but nonetheless it would nice if it automatically switched to hostpath.

Maybe using type: auto (default?)

kind: StorageClass
apiVersion: storage.k8s.io/v1
parameters:
  type: auto

When the zfs-provisioner runs it does not set the ownership information (uid/gid) on datasets that get created and this can cause pods running as non-root users to get permission errors writing to mounted volumes.

Is it currently possible to specify UID/GID that should be applied to datasets upon creation? If not, it would be useful to be able to specify them either at the PVC level or as a default via the storageclass.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• Update module k8s.io/client-go to v1

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository