Investigate options for large file storage about scikit-hep-testdata HOT 7 CLOSED

If EOS had an HTTP front-end (e.g. for eospublic.cern.ch), that would be a possibility as well.
After all, it is supported by EOS.

from scikit-hep-testdata.

It would also be useful to find a reasonable way to define a "large" file. How many MBs before we consider the file "large"? 25MB is a typical value in my experience where other services (eg my Thunder mail client, github itself, I think) starts to raise warnings. Is that large enough to handle most test files though? Need to investiage a bit...

from scikit-hep-testdata.

So, short anser to HTTP EOS: it does exist, but

• https://eospublichttp.cern.ch/eos/: requires X509
• https://eospublichttp.cern.ch:8443/eos: requires kerberos auth

The only way to get a public instance (no auth) is to request it: https://cern.service-now.com/service-portal/

EDIT: if it is requested by enough people, it should be a problem

from scikit-hep-testdata.

I suppose CERN-box might be workable instead though. I think you can download files directly from that (though don't remember the last time I tried).

from scikit-hep-testdata.

I believe you'd have to pay for Git LFS and S3, FYI. As S3 is a web protocol, the main advantage over just a random website is access control (multiple people can change the files) for this. However, if you find somebody with a CEPH installation, that speaks S3. Git LFS might be the best option though since it saves changes for posterity.

from scikit-hep-testdata.

We now have a first test-instance of an EOS Web page hosting some data:
http://fast-hep-data.web.cern.ch/fast-hep-data/

The only file at the moment:

• http://fast-hep-data.web.cern.ch/fast-hep-data/cms/L1T/CMS_L1T_study.root

This space is limited to 1TB for now but still represents a useful amount.

from scikit-hep-testdata.

Closing given the discussions, decisions and actions of #48. FYI @jpivarski, @henryiii.

from scikit-hep-testdata.