gluster / gluster-subvol Goto Github PK

Gluster-fuse (upon which gluster-subvol is based) struggles with metadata-intensive and small-file workloads.

We would like to have a subvol type that in analogous to gluster-block. It needs to preserve the original objectives of gluster-subvol (i.e., large PV count, thin provisioning to allow overcommit, etc.)

Docker cloud is used to build the recycler pod image. The only tag currently supported is "latest" which is derived from master.
Augment the build rules to pick up version tags from this repo to build actual version labels so that any deployment using this image can control their version/upgrade cycle.

Connect up travis-ci.org to run shellcheck on all the scripts used in the gluster-subvol project.

This depends on cleanup work in #12 first.

Currently there is a single CRD for both the recycler and the plugin. The recycler fields additionally allow specifying an arbitrary number of volumes to recycle.

The CRD should be split into two:

• recycler: This should support recycling exactly 1 supervol
• plugin: This should deploy the flex plugin

The flex DS should have a sidecar(s) that:

• provides the flex logfile to stdout
• provides the fuse log to stdout
• properly rotates the log files to prevent space exhaustion

The current RBAC rules work for kube, but OpenShift will likely be different.

Create the CSV and other metadata necessary to have gluster-subvol-operator deployed and upgraded by OLM.

Starting point: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/Documentation/design/building-your-csv.md

Issues to consider:

• The flex DaemonSet needs access to hostPath
• Both the recycler and flex need to run w/ SCC anyuid

Having seen some documentation that Flex doesn't support RWX, the current PVs are created as ReadWriteOnce. Test whether ReadWriteMany can be used and potentially change volcreator to provision RWX volumes.

Change creator.sh to create the PV manifests with both RWO and RWX access modes so that the gluster-subvol PVs can be used as either type.

The PVs created when deploying the recycler don't have a StorageClass set, so they get tagged with the cluster default. While there are no ill effects, it can be confusing when viewing PVs in the cluster.

Change the deployment scripts to set a blank class so the default does not get assigned.

Currently, the volrecycler-sa has cluster-admin. Figure out what the real set of permissions are that it should have and adjust the sa appropriately.

gluster-subvol will only work on Linux hosts. Apply proper nodeAffinity to ensure it is only scheduled there. This also requires properly merging in other nodeAffinity specs from the CR.

Currently, the recycler pod is just a pod, so it will not get properly restarted upon node failure.

volrecycler.yml needs to be reworked so that the recycler is a Deployment, ensuring it will get respawned should the pod fail or be killed.

The recyclers and flex should be able to be controlled via node selectors and tolerations.

The mount path for the supervol, set here:
https://github.com/JohnStrunk/gluster-subvol/blob/4c482d1b7e488dab04d212e2dcec30d2add7e399/glfs-subvol/glfs-subvol#L18

may cause excessive reloads of the flex plugins due to the dynamic flex discovery. See: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/flexvolume-deployment.md#detailed-design

I believe this can be fixed by prepending the mount directory name w/ a "."

Fix the errors produced by shellcheck

• glfs-subvol/glfs-subvol
• volcreator/creator.sh
• volrecycler/locker.sh
• volrecycler/recycler.sh

Each pod should have appropriate resource requests/limits for:

• memory
• cpu
• ephemeral storage

The statically created PVs should be tagged with a storage class to make it easier to use multiple types of storage.

kubernetes/kubernetes#68680 added a new flag on the init call fsGroup that, when set to false, will skip the recursive change group of all the files in the PV on each mount.

We should return fsGroup:false from the init call to prevent the chown.

The main Gluster driver already skips it, so this change would make the behavior consistent with the main implementation of Gluster and greatly speed the mount operation for volumes with a high file count.

If the fuse process dies, pods get stuck and won't terminate. The error commonly seen is 'transport endpoint not connected'. The state that the system is left in results in kubelet being confused about the state of the mount.

Investigate (and implement) whether auto_unmount will provide better error recovery.

Currently, they are available at:

• registry.access.redhat.com/ubi7-dev-preview/ubi-minimal
• registry.access.redhat.com/ubi8-dev-preview/ubi-minimal

We should use ubi8

Currently, the recycler has cluster IP and volume name passed in as arguments, and these must match the PV/PVC to which it gets attached. If an admin misconfigures this, there is a possibility of either data loss (wrong directory being wiped) or data leakage (failing to wipe the directory).

volcreator could be enhanced to annotate the root directory with the cluster IP and volume name such that we avoid having to pass it to the recycler. On startup, the recycler could read these parameters and use them when looking up the PVs.

This would reduce the amount of configuration as well as improve safety.

The recycler needs to get bound to a specific PV, and that PV must never get claimed by any other PVC.

Currently, I'm using a unique storage class name as a work-around, but this is incorrect. I should be using volumeName in the PVC and claimRef in the PV.

https://docs.openshift.org/latest/dev_guide/persistent_volumes.html#persistent-volumes-volumes-and-claim-prebinding

We should add current timestamps to the log file in addition to the existing invocation timestamps so that we can tell if the script execution is slow for some reason.

I propose making the change to the log function to add the new timestamp as the first field on the line, moving the current one to 2nd. This is more in keeping w/ a standard logging style.