Feature request: ability to generate resulting secrets in a file format about kubernetes-external-secrets HOT 22 CLOSED

@silasbw What about applications that we have no control over? For instance, Argo CD Notifications expects a stringData secret in a similar format:

data:
  notifiers.yaml: |
    slack:
      token: <secret token>

How do you recommend this is handled?

from kubernetes-external-secrets.

Let's get @darose's feedback on my proposal, but we're going to make this change regardless.

from kubernetes-external-secrets.

In your example @darose, my suggestion would be to split your dev-env-credentials by Application A and B (in some sense, the arrangement you layout also violate least privilege).

Regarding your proposal for transforming/filtering/plucking results from backends (e.g., AWS SM) before writing them to a frontend (e.g., a Secret object), I think you're on to something that could be more general than supporting AWS SM. Like, maybe someday there's an HTTPS backend and I can imagine it would be really useful to pluck specific values from the response before writing them to a frontend.

My worry is: 1. I don't have enough experience to understand what a good longer term direction is; and 2. if we do the wrong thing then we're stuck supporting/maintaining it.

With that, you can:

1. sit tight and eventually we'll get enough experience and anecdotal data to make an informed decision; or
2. if you're willing to invest time to contribute, write up a more detailed proposal that considers uses with other types of backends and we can discuss that.

Thoughts?

from kubernetes-external-secrets.

I think the new templating feature would cover these needs? 🎉

from kubernetes-external-secrets.

kong secret also requires both data and stringData

apiVersion: v1
kind: Secret
metadata:
  name: kong-secret
  namespace: monitoring
stringData:
  kongCredType: basic-auth
  username: user
data:
  password: pass

from kubernetes-external-secrets.

@darose can you describe better why "This isn't really ideal from a usability standpoint" ?

FYI, you can already accomplish this easily. Just create your "config.yaml" secret in secret manager directly.

from kubernetes-external-secrets.

Result would be:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  config.yaml: <base64encoded>

from kubernetes-external-secrets.

Does it make sense @darose? If yes, I will close the issue.

from kubernetes-external-secrets.

When I say it isn't ideal from a usability standpoint, I mean that in the sense that it's preferable to provide an application with a single file of secrets containing multiple key value pairs, rather than multiple files in a directory, each containing a single value.

As far as 'Just create your "config.yaml" secret in secret manager directly', sounds like I need to get a bit more specific about our use case. Yes, that is technically possible, though not in the way we've been using ASM. (Or would like to use it.) Yes, one can create a "plaintext" secret in ASM that contains a big chunk of secrets data (e.g., multiple key-value settings, or some JSON). However, we've been trying to stick with using straight key/value secrets in ASM. In that scenario, it is not possible to have all of the k/v pairs get written to a secrets object in such a way that they can be mounted into a container as a single file.

For example, using this external secrets definition:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: demo
  namespace: default
secretDescriptor:
  backendType: secretsManager
  data:
  - key: demo/demo-secret
    property: user
    name: connection.user
  - key: demo/demo-secret
    property: password
    name: connection.password

... we wind up with this secrets object:

apiVersion: v1
kind: Secret
metadata:
  name: demo
  namespace: default
type: Opaque
data:
  connection.user: <...encoded user...>
  connection.password: <...encoded password...>

When mounted inside a container, this will result in 2 separate files, connection.user and connection.password.

What would be much more useful would be if the external secrets definition I posted above could generate a secrets object that looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: demo
  namespace: default
type: Opaque
stringData:
  secrets.txt: <...encoded blob containing k/v pairs for connection.user and connection.password...>

Or, similarly, if it could generate a secrets.json file, containing the k/v pairs in json format.

from kubernetes-external-secrets.

I understand but this seems a very specific use case to me.
Generally, I'd like to stay as much transparent and minimal as possible at controller level:

1. if we can leverage backend data structure itself we should
2. do one thing only, and do it well

Finally, Secrets Manager supports both plain text and key/value secrets. You should chose one, and reflect that choice in your application.

To summarize, I think this is not relevant to the core of the project itself. It looks "unexpected" and not following the "kubernetes" standards: it is sort of "transformation feature" that happens between AWS Secrets Manager (backend) and Kubernetes (Secret)".

That said, I am going to sleep on it. I'd like the option of @silasbw here as well.

from kubernetes-external-secrets.

Fair enough. FYI, in terms of a more concrete suggestion, I guess what I'm recommending is support for something like this (note the "output" section):

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: demo
  namespace: default
secretDescriptor:
  backendType: secretsManager
  output:
    name: secrets.txt
    format: text
  data:
  - key: demo/demo-secret
    property: user
    name: connection.user
  - key: demo/demo-secret
    property: password
    name: connection.password

Or this:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: demo
  namespace: default
secretDescriptor:
  backendType: secretsManager
  output:
    name: secrets.json
    format: json
  data:
  - key: demo/demo-secret
    property: user
    name: connection.user
  - key: demo/demo-secret
    property: password
    name: connection.password

from kubernetes-external-secrets.

@darose What do you suggest the "output" should look like to indicate the current behavior?

from kubernetes-external-secrets.

I gave examples in the code I posted in my comment #73 (comment)

from kubernetes-external-secrets.

What if we changed changed the behavior of the secretsManager backend to conditionally parse the AWS SM value if (and only if) the data object does not specify a property value? Right now we unconditionally parse here. This would allow users to do something like:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: demo
  namespace: default
secretDescriptor:
  backendType: secretsManager
  output:
    name: secrets.json
    format: json
  data:
  - key: demo/demo-secret
    name: connection.json

connection.json would hold the entire k/v contents of demo/demo-secret. It's not exactly what you're asking for @darose (e.g., you're potentially getting superfluous secret data), but I think it addresses some of the concerns / objections @JacopoDaeli brings up. Would it work for you?

We should probably make this change regardless.

from kubernetes-external-secrets.

Right @silasbw this makes sense to me. Re: We should probably make this change regardless., I agree.

What about the output option? Does it becomes irrelevant right?
If we do not specifies the property value, we do not parse secretValue, therefore the secret data (stored in SecretsManager) is considered as "plain text", and it's converted as Secret property call "connection.json".

Finally, if you mount the Secret as volume, you would end up having /secrets/connection.json file.

What do you think @silasbw ?

from kubernetes-external-secrets.

Thanks much for giving this consideration @silasbw @JacopoDaeli . As you mentioned, it's not an exact match with what I had requested. (As you mentioned, there's the concern of superfluous secret data.) But it might be sufficient for our purposes. I have to think through how big a security concern the superfluous secret data might be.

Just to clearly lay it out, a fully fleshed-out use case for our team's ideal solution would be something like so:

• Store a bunch of k/v pairs into a secret in ASM, say, such as "dev-env-credentials":

SharedDbUsername:  foo
SharedDbPassword:  bar
ExternalSystem1Username:  john.doe
ExternalSystem1Password:  blahblahblah
ExternalSystem2Username:  jane.doe
ExternalSystem2Password:  mehmehmeh
ExternalSystem3Username:  lady.gaga
ExternalSystem3Password:  gagagaga

• Given a requirement that application A needs access to the shared db and system 1, generate a secrets object containing only the k/v pairs SharedDbUsername, SharedDbPassword, ExternalSystem1Username, and ExternalSystem1Password, and have that get mounted into application A as conf/credentials.json (or .txt).

• Given a requirement that application B needs access to the shared db and system 2, generate a secrets object containing only the k/v pairs SharedDbUsername, SharedDbPassword, ExternalSystem2Username, and ExternalSystem2Password, and have that get mounted into application B as conf/credentials.json (or .txt).

What you're proposing comes very close to this. The obvious issue of course is that it'd be violating "principle of least privilege", since applications would be provided with secret credentials they don't need. (And arguably shouldn't have.)

I guess what I had been thinking of as a solution was that it might be possible to be a bit smarter about how the external secret object generates the yaml for the k8s secret. So, for example, at the place where you generate the secret yaml maybe it would be possible to use "stringData" instead of "data", and then just write the desired k/v pairs into a customized secrets object that contains only the desired data, like I described in my previous comments.

That said, this is obviously my ideal solution (and, I think, a good use case) but I'll admit I'm not sure if it is for anyone else as well. Please let me know what you decide. If this isn't something you guys have the time/inclination to tackle, would this be something you might be open to merging in if we were to provide a code contribution? If so, I can look into whether our team might be able to contribute some coding time/resources to making this happen.

from kubernetes-external-secrets.

Thanks for clarifying this @darose . Unfortunately, I would not implement this at this time. @silasbw what do you think?

from kubernetes-external-secrets.

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

from kubernetes-external-secrets.

What I ended up doing was this:

---
apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: argocd-notifications-secret
  namespace: argo
spec:
  backendType: vault
  vaultMountPoint: production-eks-cluster
  vaultRole: external-secrets-production
  kvVersion: 1
  data:
  - name: notifiers.yaml
    key: secret/kubernetes/argo/slack
    property: notifiers.yaml

And the secret in Vault looks like:

slack:
  token: <Slack Token>
  username: argocd
  icon: https://avatars3.githubusercontent.com/u/30269780?s=60&v=4

from kubernetes-external-secrets.

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

from kubernetes-external-secrets.

This issue was closed because it has been stalled for 30 days with no activity.

from kubernetes-external-secrets.

I think that this method solve the problem 😄 , in this example the secrets value will be converted to the dotenv format, you can change the template parse for the format that you need

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  backendType: secretsManager
  template:
    stringData:
      dotenv: |
        <%= Object.entries(JSON.parse(data.s1)).map(e => e[0] + "=" + "'" + e[1] + "'").join("\n") %>
    metadata:
      annotations:
        reloader.stakater.com/match: "true"
  data:
    - key: /app/secrets
      name: s1

from kubernetes-external-secrets.