Need Guidance on S3FS / CVMFS about galaxy-helm HOT 3 OPEN

Hi @NilsKrattinger We would recommend continuing to use CVMFS, for several reasons.

1. In general, cvmfs has been the main reference data source used by the galaxy community.
2. We considered alternatives because there was a period in which the csi for cvmfs in particular lacked maintenance. This has been resolved now, and we are seeing regular updates.
3. s3-csi (and alternatives like Yandex), does not perform as well as cvmfs when dealing with lots of small files. This is because each file requires a request, whereas cvmfs uses a larger block based format.
4. The above + the fact that cvmfs is generally optimized for read-only data, and the s3 alternatives are not, resulted in worse performance for refdata. We observed a dramatic increase in startup times after switching to it. Possible future optimizations may be able to remedy this to some degree, but at least for the time being, it's the slower option.

For these reasons, we have switched back to cvmfs as the refdata repository of choice. Ideally however, the choice should not matter, as it should (at least in principle) be trivially possible to switch from one to the other just by enabling/disabling the corresponding option, and the chart will just use the new source.

from galaxy-helm.

@nuwang Thanks for sharing the reasons and story behind this, really appreciate it!

As for the low-coupling between refdata and the storage solution, i fully agreed that the CSI / Storage solution should ideally not matter, as long as it support ROX operations.

Going down that road, would something like a refdata.storageCalss attribute in Values.yaml be desirable?
To not break compatibility i think that, this would have to by override by cvmfs.enable and S3FS.enbale, or it could be defaulted to cvmfs / s3fs when left empty?

from galaxy-helm.

Yes, the storage class can currently be specified and it will be used when creating the pvc:

However, I just realised that changing the storageclass on a pvc midstream is not supported by k8s, so we should probably add a feature to allow the pvc itself to be injected externally. When done, it should be possible to switch the pvc transparently - with running jobs continuing to use the old pvc, and new jobs just switching to the new pvc.

from galaxy-helm.