Comments (9)
ksandha
commented on May 18, 2024
1
100 PVC are getting created successfully But failing on a large scale volume creation , hitting the bug gluster/glusterd2#1364
Closing the bug and updating the above mentioned bug.
from gluster-csi-driver.
rmadaka
commented on May 18, 2024
Adding some are observations here,
I tried to reproduce this issues on one more setup.
- Tried create 99 pvcs using python script.
- It took around 30 mints to bound the 17 PVCs, remaining 83 pvcs are pending state.
- After around 45 mints etcd pods status started becoming completed state.
Here each pvc size 2 GB , is it really take that much time to bounding 17 pvcs.
When i tried first time each pvc size 1Gb
Note: This issue is not only because of etcd. when etcd pods were running also, pvcs were taking too much time to getting bound..
from gluster-csi-driver.
JohnStrunk
commented on May 18, 2024
Please check for errors in the logs of the CSI provisioner pod and the gluster pods.
from gluster-csi-driver.
Madhu-1
commented on May 18, 2024
@rmadaka if you are thinking the issue is with csi-driver please provide the logs from all CSI pods
pod/csi-attacher-glusterfsplugin-0 2/2 Running 0 7h49m
pod/csi-nodeplugin-glusterfsplugin-6fgq8 2/2 Running 0 7h49m
pod/csi-nodeplugin-glusterfsplugin-hlb95 2/2 Running 0 7h49m
pod/csi-nodeplugin-glusterfsplugin-pmhvd 2/2 Running 0 7h49m
pod/csi-provisioner-glusterfsplugin-0 2/2 Running 0 7h49m
pod/etcd-bq58l8bfkz 0/1 Completed 0 7h52m
pod/etcd-operator-7cb5bd459b-97hxq 1/1 Running 0 7h52m
pod/etcd-phfpxjxg6n 0/1 Completed 0 7h50m
pod/etcd-wfkrgtvhwb 0/1 Completed 0 7h51m
pod/gluster-kube1-0 0/1 CrashLoopBackOff 39 7h50m
pod/gluster-kube2-0 1/1 Running 39 7h50m
pod/gluster-kube3-0 1/1 Running 0 7h50m
i feel the issue is with the glusterd2 pods and ETCD pods. as all the ETCD pods are in completed state, glusterd2 needs ETCD to store its information during the volume creation.
from gluster-csi-driver.
rmadaka
commented on May 18, 2024
CSI-Provisioner Logs:
Below kind of logs i found for all pending pvcs
E1015 21:27:31.290877 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc84", provisioner will not attempt retries for this claim
E1015 21:27:31.290893 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc95", provisioner will not attempt retries for this claim
E1015 21:27:31.290944 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc29", provisioner will not attempt retries for this claim
E1015 21:27:31.290986 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc37", provisioner will not attempt retries for this claim
E1015 21:27:31.291057 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc45", provisioner will not attempt retries for this claim
E1015 21:27:31.291084 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc77", provisioner will not attempt retries for this claim
E1015 21:27:31.291115 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc90", provisioner will not attempt retries for this claim
E1015 21:27:31.291134 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc18", provisioner will not attempt retries for this claim
E1015 21:27:46.289657 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc50", provisioner will not attempt retries for this claim
E1015 21:27:46.289725 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc66", provisioner will not attempt retries for this claim
E1015 21:27:46.289739 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc69", provisioner will not attempt retries for this claim
E1015 21:27:46.289761 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc23", provisioner will not attempt retries for this claim
E1015 21:27:46.289773 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc24", provisioner will not attempt retries for this claim
E1015 21:27:46.289784 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for ```
Note: From gluster-provisioner didnt find any kind of logs
from gluster-csi-driver.
JohnStrunk
commented on May 18, 2024
E1015 21:27:31.290877 1 controller.go:638] Exceeded failedProvisionThreshold threshold: 15, for claim "default/gcs-pvc84", provisioner will not attempt retries for this claim
We need to take a look earlier than this... kube-csi has given up, but we need to figure out why it needed to retry at all. My current guess is that either gd2 pods couldn't create volumes (beyond the initial 17) or it was taking a long time to do so, and we're not handling that well in the CSI driver.
from gluster-csi-driver.
vpandey-RH
commented on May 18, 2024
@rmadaka Can you please check GD2 logs once ?
from gluster-csi-driver.
atinmu
commented on May 18, 2024
Is this still seen in latest testing?
from gluster-csi-driver.
Madhu-1
commented on May 18, 2024
@ksandha can you please verify this issue?
from gluster-csi-driver.
Related Issues (20)
- REST spec HOT 2
- Can't start glusters daemonset on host, that already runs rpcbind
- cloud.docker.com builds fail intermittently due to lint issues
- Volume size mismatch between pvc and gluster when a decimal point pvc is created HOT 6
- Dependency mess on kubernetes-csi/drivers and kubernetes-csi/csi-tests repos HOT 8
- update side car containers to v1.0.1 HOT 1
- Non-root is unable to write to PV HOT 1
- don't return UNKNOWN in capabilities HOT 2
- thin-arbiter volume type support HOT 1
- Add to readme GD2 example HOT 1
- push loopback driver to dockerhub HOT 5
- Fix gocyclo nolint in controllerserver.go
- Failing to build images on master banch HOT 2
- Rotate logs for glusterfs fuse mounts
- virtblock mounts as fuse HOT 4
- Block volume mount fails if no loop devices are present on the host HOT 2
- Per volume encryption
- Incorrectly named drivers
- Is Kubernetes support going away? HOT 6
- Failed to connect to glusterfs 10.4： connection reset by peer
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gluster-csi-driver.