Creation of `ReatWriteOnce` fails with `mounter.SetupAt failed: rpc error: code = Unknown desc = Exception calling application: [32] b` about kadalu HOT 13 CLOSED

@papanito we made changes in the image os, and with that the tool versions used for xfs etc.

Please see if 0.7.6 is any better for you. We will close this issue for now, as we have not seen the particular issue in long time now. If it exists with 0.7.6 please feel free to reopen. Thanks for patience 🙏

from kadalu.

Is this consistent with RWO PVC ? Because I too had an issue with RWO when I tried, but noticed thats because of failing to mount the hostvolume, so, the file was not accessible on app pods.

When this happens, would need log of csi-nodeplugin-***** (in kadalu namespace), go with --all-containers option for logs. There were no issues in data path of RWO pv.

from kadalu.

Here you go. I've deleted the installation (helm delete) and the reinstalled it with ReadWriteOnce.
cis-nodeplugin.log

from kadalu.

RWO seems the same issue cis-nodeplugin.log

from kadalu.

Here the log of storage-pool-pod: storage-pool.log

from kadalu.

Interesting. The issue here seems to be related to disconnects happening when fd_cleanup happens. Still need to debug (as the same issue is not happening for me yet). Looks like we need to handle #75 for this same reason. At present this is a glusterfs disconnect between client and server.

from kadalu.

Is there a way I can debug it or maybe give you more details?
And why does RWX work and the others not?

from kadalu.

I am suspecting the issue to be similar to https://lists.gluster.org/pipermail/gluster-users/2020-January/037557.html (in case of RWO). I recommend to use direct-io-mode=disable for mount point. This should help in applications which would need DB like access.

from kadalu.

Still had not the chance to test replica3. Path requires xfs as underlying fs to simulate 3 nodes but I have ext4

from kadalu.

If you could make a path to export, I recommend creating a file truncate -s Ng $path/disk-image and use that as device: $path/disk-image in the storage config file. Again, this is good only for testing.

Ref: https://github.com/kadalu/kadalu/blob/master/tests/minikube.sh#L153..L154

from kadalu.

I've completely removed kadalu by running the cleanup.sh script and the used replica3. Here the logs trying a replica3. I'm actually using 3 nodes
csi-nodeplugin-m8phv.log
csi-provisioner-0.log
kadalu.events.log
server-storage-pool-main-0-node001-0.log
server-storage-pool-main-1-node002-0.log
server-storage-pool-main-2-node003-0.log
csi-nodeplugin-qmgzg.log
csi-nodeplugin-w6dfz.log

The error is at the moment different, I assume cause the cleanup was not properly

E0127 12:14:01.463808       1 controller.go:700] error syncing claim "prometheus/prometheus-server": failed to provision volume with StorageClass "kadalu.replica3": rpc error: code = Unknown desc = Exception calling application: [1] b'' b'mkfs.xfs: /mnt/storage-pool-main/virtblock/b6/ce/pvc-d95bf5f5-2629-43ba-ba37-0ae57778b4f6 appears to contain an existing filesystem (xfs).\nmkfs.xfs: Use the -f option to force overwrite.'
I0127 12:14:01.463802       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-server", UID:"d95bf5f5-2629-43ba-ba37-0ae57778b4f6", APIVersion:"v1", ResourceVersion:"5363801", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "kadalu.replica3": rpc error: code = Unknown desc = Exception calling application: [1] b'' b'mkfs.xfs: /mnt/storage-pool-main/virtblock/b6/ce/pvc-d95bf5f5-2629-43ba-ba37-0ae57778b4f6 appears to contain an existing filesystem (xfs).\nmkfs.xfs: Use the -f option to force overwrite.'

from kadalu.

Thank you for your contributions.
Noticed that this issue is idle since 180 days! There is a possibility that this issue is already fixed in later releases. Please upgrade and check! If I don't hear any update in this issue in next 2 weeks, will be closing the issue.
That doesn't mean one can't re-open the issue! Just comment on the issue, and click 'Reopen', if you still have the issue.

from kadalu.

@amarts will this be fixed?

from kadalu.