Private buckets support in Rafter about rafter HOT 19 CLOSED

Generally speaking, it is doable and we can reuse solutions already available on the market, but first lets me explain what the problem is.

MinIO is supporting two ways of authentication, S3 like signatures, when communicating via REST, and basic when using MinIO client. S3 like signature is an encrypted (with secret key) string that contains operation type, content type, date, and resource:

signature_string="GET\n\n${content_type}\n${date}\n${resource}"
signature=`echo -en ${signature_string} | openssl sha1 -hmac ${s3_secret} -binary | base64`

It is then put to Authorization header together with the access key:

Authorization: AWS ${access_key}:${signature}

And the problem is that we don't have ready to use proxies that will support AWS authorization, but it doesn't mean that it is not possible to use them and we should create our own.

We should simply add an additional proxy between MinIO and some authentication proxy. That addition proxy will be authenticated to MinIO and will pass all requests, so the communication will look like that:

-----------        --------------       ---------------       ---------
| Browser |  ----> | Auth proxy | ----> | MinIO proxy | ----> | MinIO |
-----------        --------------       ---------------       ---------

Where:

• Auth proxy is a proxy responsible for authentication
• MinIO proxy is a lightweight proxy authenticated to MinIO (with client or signature)

Communication between Auth proxy, MinIO proxy and MinIO is on localhost as all three are on the same pod because they work as sidecars.

As an Auth proxy I would use Keycloak as it is designed for that.

MinIO proxy is something that we will need to implement, but it is relatively easy to do as MinIO has a library for Go.

The other benefit of having an additional proxy in the middle is the ability to add additional authorization layers in the future to verify if a particular user should have access to the resource.

Ok, but how to provide access to public buckets when communication goes via proxies? We can simply add additional sidecar responsible for routing, let's call it router:

-----------       ----------  private  --------------       ---------------       ---------
| Browser | ----> | Router | --------> | Auth proxy | ----> | MinIO proxy | ----> | MinIO |
-----------       ----------           --------------       ---------------       ---------
                      |                             public                            |
                      |---------------------------------------------------------------|

Router is responsible for checking if access is public or private and route to MinIO or Auth proxy. It is on the same pod as proxies and MinIO and communicates with them on the localhost. Router will read the access policy from Bucker CR.

from rafter.

@derberg @magicmatatjahu updated: https://github.com/kyma-project/kyma/issues/4024#issuecomment-560259481

If Router sits in a container in the same pod as Minio, this is all fine. So he will simply query for the assets, match the bucket base url with given asset, check the referenced bucket policy and act accordingly? Makes sense.

It can read from Bucket CR without checking Asset CR. Everything is in URL 😄

from rafter.

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

from rafter.

First we want to do a design proposal for 1.7 to make sure what has to be done

from rafter.

@piotrmsc as you have experience with Keycloak and authentication/authorization can you take a look at https://github.com/kyma-project/kyma/issues/4024#issuecomment-560259481 ?

from rafter.

@michal-hudy just to specify, those proxies would work as sidecars in the same POD with minio + I'm curious if MinIO proxy cannot be done with Istio and lua script instead of writing new component

from rafter.

@derberg sidecars part updated.

MinIO proxy probably may be implemented as lua script, but as I remember it is not nice to use and is hard to debug. Also, I try to propose something that is independent of infrastructure because Rafter can work standalone without Kyma and Istio. And, really, creating MinIO proxy in Go is super easy (few lines of code), and it can be released with Rafter.

And of course, if it is recommended in Kyma, we can replace it with lua...

from rafter.

@michal-hudy you are right, I forgot that lua and istio would bring in strong dependency to Istio which is not ok for Rafter as standalone

from rafter.

@michal-hudy Congrats! Great investigation! I understand solution, but I only have a problem with providing access to public buckets. Two endpoints is necessary? Can't we use proxies for that, in meaning, if bucket is private then authenticate, if not, send request without checking? As far as I know, file's url from asset is concatenating parts of bucket, asset and appropriate file, so checking if bucket is private or not from request will be easy. Two endpoints from my side is a huge problem, because if someone will want use Rafter, only from front-end app, then this person will have to create unnecessary backend to switch to private or public buckets. What do you think?

from rafter.

@magicmatatjahu it is possible to have one endpoint, but we will need to implement everything by ourselves or add additional proxy before Auth proxy that will point to the MinIO for public buckets and to the Auth proxy for private. So, when we add an additional proxy, the flow will look like that:

-----------       ----------  private  --------------       ---------------       ---------
| Browser | ----> | Router | --------> | Auth proxy | ----> | MinIO proxy | ----> | MinIO |
-----------       ----------           --------------       ---------------       ---------
                      |                             public                            |
                      |---------------------------------------------------------------|

Router is responsible for checking if access is public or private and route to MinIO or Auth proxy

Thanks to that we will have one endpoint.

Btw. in flow without router access to public buckets is still possible via secure endpoint but you need to be authenticated.

from rafter.

Can @magicmatatjahu elaborate more on why it is a problem to have subdomains

minio.rafter.dev
private.minio.rafter.dev

or endpoints

minio.rafter.dev
minio.rafter.dev/private

The only valid reason for me would be an argument that:

• AWS and others also do not have access level specified in the URL
• switching bucket from private to the public would cause base url change for the bucket
  Am I discussing it here with myself? :)

@michal-hudy you do not need any Router. User/backend should by default read information about Bucket CR before it fetches data references in the Asset CR. In other words, the current CBS component, for example, would need to have a query for assets extended with an ability to ask for a referenced Bucket to check the spec.policy value

from rafter.

@michal-hudy you do not need any Router. User/backend should by default read information about Bucket CR before it fetches data references in the Asset CR. In other words, the current CBS component, for example, would need to have a query for assets extended with an ability to ask for a referenced Bucket to check the spec.policy value

Why? We can have everything built-in in Rafter, so the user will not have to care about the endpoint and checking the policy. MinIO will be represented by one endpoint storage.rafter.dev and changing the policy will not change it. So if you call https://storage.rafter.dev/test/file.bin where test is a public bucket it will work and then you switch test to private access then the URL will be the same but it will require token, because if not provided Keycloak will redirect you to the login page. From UI you can always make calls with token, you don't have to check if the bucket is private or public. What is important, router will be on the same pod as proxies and MinIO so all communication between them will be on the localhost.

minio.rafter.dev/private

I don't recommend it, as you can have a bucket with private as a name.

So yes, I agree with @magicmatatjahu that having one endpoint is much more user-friendly than splitting it to the endpoint with authentication and not.

One more reason for having one endpoint, that will not change with bucket policy change is links sharing.

Edit: We only need to remember that the URL contains a remote bucket name, not CR. And we are supporting cluster and namespace buckets, but this is a technical detail that doesn't block one endpoint solution, we will just need to have a proper index on the informer.

from rafter.

Yes, definitely the only way to go here is on single minio.rafter.dev so the URLs do not switch on bucket privacy change. This is obvious and I guess we all agree so you can update the proposal.

If Router sits in a container in the same pod as Minio, this is all fine. So he will simply query for the assets, match the bucket base url with given asset, check the referenced bucket policy and act accordingly? Makes sense.

We only need to remember to not make modifications to the minio chart with those containers and maybe consider adding an admission webhook that will mutate minio deployment and add our custom containers?

from rafter.

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

from rafter.

This issue has been automatically closed due to the lack of recent activity.

from rafter.

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

from rafter.

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

from rafter.

This issue has been automatically closed due to the lack of recent activity.

from rafter.

This feature will not be worked on by kyma contributors as we don't see value in this anymore.
We no longer want to host (public or private) assets in the kyma runtime and plan to move that feature to central control plane

from rafter.