Volume Expander Operator

The purpose of the volume-expander-operator is to expand volumes when they are running out of space. This is achieved by using the volume expansion feature.

The operator periodically checks the kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes published by the kubelets to decide when to expand a volume. Notice that these metrics are generated only when a volume is mounted to a pod. Also the kubelet takes a minute or two to start generating accurate values for these metrics. The operator accounts for that.

This operator works based on the following annotations to PersistentVolumeClaim resources:

Annotation	Default	Description
`volume-expander-operator.redhat-cop.io/autoexpand`	N/A	if set to "true" enables the volume-expander-operator to watch on this PVC
`volume-expander-operator.redhat-cop.io/polling-frequency`	`"30s"`	How frequently to poll the volume metrics. Express this value as a valid golang Duration
`volume-expander-operator.redhat-cop.io/expand-threshold-percent`	`"80"`	the percentage of used storage after which the volume will be expanded. This must be a positive integer.
`volume-expander-operator.redhat-cop.io/expand-by-percent`	`"25"`	the percentage by which the volume will be expanded, relative to the current size. This must be an integer between 0 and 100
`volume-expander-operator.redhat-cop.io/expand-up-to`	MaxInt64	the upper bound for this volume to be expanded to. The default value is the largest quantity representable and is intended to be interpreted as infinite. If the default is used it is recommend to ensure the namespace has a quota on the used storage class.

Note that not all of the storage driver implementations support volume expansion. It is a responsibility of the user/platform administrator to ensure that storage class and the persistent volume claim meet all the requirements needed for the volume expansion feature to work properly.

This operator was tested with OCS, but should work with any other storage driver that supports volume expansion.

Deploying the Operator

This is a cluster-level operator that you can deploy in any namespace, volume-expander-operator is recommended.

It is recommended to deploy this operator via OperatorHub, but you can also deploy it using Helm.

Multiarch Support

Arch	Support
amd64	✅
arm64	✅
ppc64le	✅
s390x	✅

Deploying from OperatorHub

Note: This operator supports being installed disconnected environments

If you want to utilize the Operator Lifecycle Manager (OLM) to install this operator, you can do so in two ways: from the UI or the CLI.

Deploying from OperatorHub UI

If you would like to launch this operator from the UI, you'll need to navigate to the OperatorHub tab in the console.Before starting, make sure you've created the namespace that you want to install this operator to with the following:

oc new-project volume-expander-operator

Once there, you can search for this operator by name: volume expander operator. This will then return an item for our operator and you can select it to get started. Once you've arrived here, you'll be presented with an option to install, which will begin the process.
After clicking the install button, you can then select the namespace that you would like to install this to as well as the installation strategy you would like to proceed with (Automatic or Manual).
Once you've made your selection, you can select Subscribe and the installation will begin. After a few moments you can go ahead and check your namespace and you should see the operator running.

Deploying from OperatorHub using CLI

If you'd like to launch this operator from the command line, you can use the manifests contained in this repository by running the following:

oc new-project volume-expander-operator

oc apply -f config/operatorhub -n volume-expander-operator

This will create the appropriate OperatorGroup and Subscription and will trigger OLM to launch the operator in the specified namespace.

Deploying with Helm

Here are the instructions to install the latest release with Helm.

oc new-project volume-expander-operator
helm repo add volume-expander-operator https://redhat-cop.github.io/volume-expander-operator
helm repo update
helm install volume-expander-operator volume-expander-operator/volume-expander-operator

This can later be updated with the following commands:

helm repo update
helm upgrade volume-expander-operator volume-expander-operator/volume-expander-operator

Metrics

Prometheus compatible metrics are exposed by the Operator and can be integrated into OpenShift's default cluster monitoring. To enable OpenShift cluster monitoring, label the namespace the operator is deployed in with the label openshift.io/cluster-monitoring="true".

oc label namespace <namespace> openshift.io/cluster-monitoring="true"

Testing metrics

export operatorNamespace=volume-expander-operator-local # or volume-expander-operator
oc label namespace ${operatorNamespace} openshift.io/cluster-monitoring="true"
oc rsh -n openshift-monitoring -c prometheus prometheus-k8s-0 /bin/bash
export operatorNamespace=volume-expander-operator-local # or volume-expander-operator
curl -v -s -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://volume-expander-operator-controller-manager-metrics.${operatorNamespace}.svc.cluster.local:8443/metrics
exit

Development

Running the operator locally

Note: this operator build process is tested with podman, but some of the build files (Makefile specifically) use docker because they are generated automatically by operator-sdk. It is recommended remap the docker command to the podman command.

export repo=raffaelespazzoli
docker login quay.io/$repo
oc new-project volume-expander-operator
oc project volume-expander-operator
tilt up

Test helm chart locally

Define an image and tag. For example...

export imageRepository="quay.io/redhat-cop/volume-expander-operator"
export imageTag="$(git -c 'versionsort.suffix=-' ls-remote --exit-code --refs --sort='version:refname' --tags https://github.com/redhat-cop/volume-expander-operator.git '*.*.*' | tail --lines=1 | cut --delimiter='/' --fields=3)"

Deploy chart...

make helmchart IMG=${imageRepository} VERSION=${imageTag}
helm upgrade -i volume-expander-operator-local charts/volume-expander-operator -n volume-expander-operator-local --create-namespace

Delete...

helm delete volume-expander-operator-local -n volume-expander-operator-local

Building/Pushing the operator image

export repo=raffaelespazzoli #replace with yours
docker login quay.io/$repo
make docker-build IMG=quay.io/$repo/volume-expander-operator:latest
make docker-push IMG=quay.io/$repo/volume-expander-operator:latest

Deploy to OLM via bundle

make manifests
make bundle IMG=quay.io/$repo/volume-expander-operator:latest
operator-sdk bundle validate ./bundle --select-optional name=operatorhub
make bundle-build BUNDLE_IMG=quay.io/$repo/volume-expander-operator-controller-bundle:latest
docker push quay.io/$repo/volume-expander-operator-controller-bundle:latest
operator-sdk bundle validate quay.io/$repo/volume-expander-operator-controller-bundle:latest --select-optional name=operatorhub
oc new-project volume-expander-operator
oc label namespace volume-expander-operator openshift.io/cluster-monitoring="true"
operator-sdk cleanup volume-expander-operator -n volume-expander-operator
operator-sdk run bundle --install-mode AllNamespaces -n volume-expander-operator quay.io/$repo/volume-expander-operator-controller-bundle:latest

Testing

Manual tests

oc new-project volume-expander-operator-test
oc apply -f ./test/volume.yaml -n volume-expander-operator-test
oc apply -f ./test/deployment.yaml -n volume-expander-operator-test

Releasing

git tag -a "<tagname>" -m "<commit message>"
git push upstream <tagname>

If you need to remove a release:

git tag -d <tagname>
git push upstream --delete <tagname>

If you need to "move" a release to the current main

git tag -f <tagname>
git push upstream -f <tagname>

Cleaning up

operator-sdk cleanup volume-expander-operator -n volume-expander-operator
oc delete operatorgroup operator-sdk-og
oc delete catalogsource volume-expander-operator-catalog

Auto expansion worked earlier then facing error "volume-expander-operator unexpected value"

Hi Team,

We are trying autoexpansion feature in our openshift environment through volume expander operator & we were getting error like
below for some of the PVCs . Yesterday expansion worked fine for one of the PVC we created but today we are getting this error & also expansion didn't take place automatically for quite some time. Later, the expansion happened automatically but the "unexpected error" event is also getting repeated.

Checked the volume expander operator logs but we don't find much in the logs to look on it further.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
annotations:
pv.kubernetes.io/bind-completed: 'yes'
pv.kubernetes.io/bound-by-controller: 'yes'
volume-expander-operator.redhat-cop.io/autoexpand: 'true'
volume-expander-operator.redhat-cop.io/expand-threshold-percent: '85'
volume-expander-operator.redhat-cop.io/expand-up-to: 5Gi
volume-expander-operator.redhat-cop.io/polling-frequency: 5m
volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
selfLink: >-
/api/v1/namespaces/test13/persistentvolumeclaims/test1360344
resourceVersion: '1037639301'
name: test1360344
uid: 93549f62-1932-48db-97e0-04dfe0a6b081
creationTimestamp: '2022-03-31T09:09:21Z'
managedFields:
- manager: kube-controller-manager
operation: Update
apiVersion: v1
time: '2022-03-31T09:09:21Z'
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations':
'f:pv.kubernetes.io/bind-completed': {}
'f:pv.kubernetes.io/bound-by-controller': {}
'f:volume.beta.kubernetes.io/storage-provisioner': {}
'f:spec':
'f:volumeName': {}
'f:status':
'f:accessModes': {}
'f:capacity': {}
'f:phase': {}
- manager: okhttp
operation: Update
apiVersion: v1
time: '2022-03-31T09:09:21Z'
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations':
.: {}
'f:volume-expander-operator.redhat-cop.io/autoexpand': {}
'f:volume-expander-operator.redhat-cop.io/expand-up-to': {}
'f:volume-expander-operator.redhat-cop.io/polling-frequency': {}
'f:spec':
'f:accessModes': {}
'f:resources':
'f:requests': {}
'f:storageClassName': {}
'f:volumeMode': {}
- manager: csi-resizer
operation: Update
apiVersion: v1
time: '2022-03-31T09:14:21Z'
fieldsType: FieldsV1
fieldsV1:
'f:status':
'f:capacity':
'f:storage': {}
- manager: manager
operation: Update
apiVersion: v1
time: '2022-03-31T09:14:21Z'
fieldsType: FieldsV1
fieldsV1:
'f:spec':
'f:resources':
'f:requests':
'f:storage': {}
- manager: Mozilla
operation: Update
apiVersion: v1
time: '2022-04-01T08:18:35Z'
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations':
'f:volume-expander-operator.redhat-cop.io/expand-threshold-percent': {}
namespace: test13-intel
finalizers:
- kubernetes.io/pvc-protection
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2560Mi
volumeName: pvc-93549f62-1932-48db-97e0-04dfe0a6b081
storageClassName: ocs-storagecluster-cephfs
volumeMode: Filesystem
status:
phase: Bound
accessModes:
- ReadWriteMany
capacity:
storage: 3Gi

Thanks,
Fayaz

redhat-cop / volume-expander-operator Goto Github PK