Minio S3 support request about skbn HOT 13 CLOSED

Hey @alexbarta,

Thanks for bringing this up!
Adding support for minio is definitely something that can be done.

Would you like to get involved and try to take this on?

from skbn.

Thanks for the fast reply!

I was already looking into setting a custom s3 configuration like in this example but with a os.Getenv approach:
https://github.com/minio/cookbook/blob/master/docs/aws-sdk-for-go-with-minio.md

However I have problems compiling my own project fork.. I'll raise a separate issue for that.

from skbn.

Thanks for helping us improve skbn!

As a guideline, try to keep things as similar as you can to current implmentation.
For example, AWS has a credentials chain (there is a link in the README file). try to avoid using the credentials environment variables explicitly.

Another example would be to try and keep the usage of --dst like it is used today. I know that to use minio you will have to provide the URL where minio can be found, but maybe there is a way to provide that as an environment variable or as a separate flag.

If you need any help, let me know :)

Good luck!

from skbn.

I have assigned the issue to you to keep things visible :)

from skbn.

got it working with latest minio! see this commit (no command line impacted :) )
alexbarta@cda0ae2

I have used the following envs:

export AWS_ACCESS_KEY_ID=minio
export AWS_SECRET_ACCESS_KEY=minio123
export AWS_S3_ENDPOINT=http://minio-svc.default.svc.cluster.local:9000
export AWS_S3_NO_SSL=true
export AWS_S3_FORCE_PATH_STYLE=true
skbn cp --src k8s://default/cassandra-0/cassandra/etc/cassandra/cassandra.yaml --dst s3://cassandra

Can you check on aws side if it is still working fine ?

from skbn.

Hey,

That's a very elegant solution!
I like it a lot.

It currently doesn't work for AWS S3. The problem is with the endpoint (shouldn't pass it if this is not minio).
Something along the lines of the following snippet should fix it for both cases:

        awsConfig := &aws.Config{}

        // ...

	if disSSL := os.Getenv("AWS_S3_NO_SSL"); disSSL != "" {
		disableSSL, err := strconv.ParseBool(disSSL)
                // handle error
		awsConfig.DisableSSL = aws.Bool(disableSSL)
	}

        // ...

        s, err := session.NewSession(awsConfig)

Would you like to create a PR for this? :)

Plus, from the example you sent (with cassandra), I assume you are interested in this feature to use cain?
https://github.com/nuvo/cain

from skbn.

Glad you like it!

And yes, I have started digging into cain (a very needed project kudos!) before finding skbn :) . Since I work on a local k8s + cassandra installation, reaching aws s3 is a bit problematic.

Btw, I have created the pull request. I will add commits to include your suggestions.

from skbn.

How do you plan to handle the ParseBool error ?

from skbn.

Actually, since ParseBool returns false for every invalid input, i don't think that we need an error handle.

Would you like to add a section about minio support in the README?

from skbn.

Sure I will

from skbn.

That was some very good work, thanks a lot!
Would you like to follow up on this and update cain as well?

from skbn.

Thanks for merging the code !
Sure no problem, I just need a couple of hours to test cain first with the new skbn.

from skbn.

Great! I'll close this issue in favor of a new issue in cain.

from skbn.