Need secrets as key and value pair available in pod environment directly about vault-k8s HOT 34 CLOSED

@pksurferdad Little late to the game, but I think your problem is your paths. You have "/secrets/dev/poc-secret", but you should have "secrets/data/dev/poc-secret".
Hope that helps.

I've been able to get this to work in a more generic fashion (i.e. you don't have to explicitly list out the env variables you want to set). Instead it creates variables based on the vault keys.
NOTE: Avoid dots in your vault keys as this doesn't play well with environment variable names.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "app-user"
        vault.hashicorp.com/agent-inject-secret-poc-secret.env: "secrets/data/dev/poc-secret"
        vault.hashicorp.com/agent-inject-template-poc-secret.env: |
          {{- with secret "secret/data/dev/poc-secret" -}}
          {{ range $Key, $Value := .Data.data }}export {{$Key}}="{{$Value}}"
          {{ end }}
          {{- end -}}
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: vault-auth
      containers:
      - name: app
        image: jweissig/app:0.0.1
        args: ["bin/sh", "-c", "source /vault/secrets/poc-secret.env"]

from vault-k8s.

@hoantran3108 Here is the command instruction that I used to do this with an influxdb image. Of course the final command will depend on the configuration of your particular image.

        command: ["/bin/bash", "-c", "source /vault/secrets/influxdb_env && ./entrypoint.sh influxd"]

from vault-k8s.

Hi @arunporwal, this is not possible with the injector today, but we are considering having an option that would generate an envfile which your container could source.

One idea could be to use custom templates to create files with a common filename structure:

   vault.hashicorp.com/agent-inject-template-db.env: |
      {{- with secret "database/creds/db-app" -}}
      #!/bin/bash
      export USERNAME={{ .Data.username }}
      export PASSWORD={{ .Data.password }}
      {{- end }}
   --- MORE SECRETS HERE ---
   ...

Then your container could source all the files: source /vault/secrets/*.env

This use case works well for static secrets.

In my case I try to use this approach with dynamic secrets but after some time the secret change and the app on that container stop working.

You guys have some suggestions on how to deal with this use case?

from vault-k8s.

Sourcing env file is not an option for distroless containers.

from vault-k8s.

Hi guys any solution?

from vault-k8s.

Hi @arunporwal, this is not possible with the injector today, but we are considering having an option that would generate an envfile which your container could source.

One idea could be to use custom templates to create files with a common filename structure:

   vault.hashicorp.com/agent-inject-template-db.env: |
      {{- with secret "database/creds/db-app" -}}
      #!/bin/bash
      export USERNAME={{ .Data.username }}
      export PASSWORD={{ .Data.password }}
      {{- end }}
   --- MORE SECRETS HERE ---
   ...

Then your container could source all the files: source /vault/secrets/*.env

from vault-k8s.

@jasonodonnell : Thanks for the update. Really appreciated.

But how to source that file in container shell before container it start. There are only 2 lifecycle container hook i believe. ( Prestop and Poststop ) both can be used after after container start or stop and there is no hook which say run it before container start.
Or if we use init container , i think that will also not worked because it will source env in init container shell only. any idea on this ?

i tried below but getting error:
Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"source\": executable file not found in $PATH": unknown

tried as :

      command:
       - source "/vault/secrets/test.env"

or

      command:
       - source 
       - "/vault/secrets/test.env"

from vault-k8s.

I think you would need to do:

command: ["/bin/sh", "-ec"]
args: ["source /vault/secrets/test.env"]

from vault-k8s.

Hello, have news to this request?

from vault-k8s.

The same question.
Do you have news?

from vault-k8s.

@jasonodonnell : Are we going to have this feature in next release ?
rather than injecting using source file.

from vault-k8s.

Related feature request: #14

from vault-k8s.

@arunporwal Did you manage to source the env file using the command, args in pod's specification? For me, using the command and args would overwrite the Dockerfile ENTRYPOINT so the container is not able to run as expected.

from vault-k8s.

@taffarel-gaivota do you have an example of how you sourced the variables in your example here #45 (comment)? Specifically, I'm not exactly sure what should be included in the <entrypoint script>

containers:
- name: web
image: alpine:latest
args: ["sh", "-c", "source /vault/secrets/config && entrypoint script>"]

from vault-k8s.

Hi @jasonodonnell , i see how this #45 (comment) is supposed to work now and i followed the instructions posted here; however, when i exec into the pod, i'm not seeing the template file in the vault/secrets volume mount and only see the fetched secret. there are no errors in the vault-agent-init logs. i also included the pod deployment. any ideas on why the template file is not being created?

vault-agent-init logs

2020-10-12T14:28:11.009Z [INFO]  sink.file: creating file sink
2020-10-12T14:28:11.009Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2020-10-12T14:28:11.009Z [INFO]  auth.handler: starting auth handler
2020-10-12T14:28:11.009Z [INFO]  auth.handler: authenticating
2020-10-12T14:28:11.009Z [INFO]  template.server: starting template server
2020/10/12 14:28:11.009896 [INFO] (runner) creating new runner (dry: false, once: false)
2020-10-12T14:28:11.010Z [INFO]  sink.server: starting sink server
2020/10/12 14:28:11.010248 [INFO] (runner) creating watcher
2020-10-12T14:28:11.020Z [INFO]  auth.handler: authentication successful, sending token to sinks
2020-10-12T14:28:11.020Z [INFO]  auth.handler: starting renewal process
2020-10-12T14:28:11.020Z [INFO]  sink.file: token written: path=/home/vault/.vault-token
2020-10-12T14:28:11.020Z [INFO]  sink.server: sink server stopped
2020-10-12T14:28:11.020Z [INFO]  sinks finished, exiting
2020-10-12T14:28:11.020Z [INFO]  template.server: template server received new token
2020/10/12 14:28:11.020499 [INFO] (runner) stopping
2020/10/12 14:28:11.020520 [INFO] (runner) creating new runner (dry: false, once: false)
2020/10/12 14:28:11.020633 [INFO] (runner) creating watcher
2020/10/12 14:28:11.020670 [INFO] (runner) starting
2020-10-12T14:28:11.063Z [INFO]  auth.handler: renewed auth token
2020/10/12 14:28:11.132482 [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/poc-secret"
2020/10/12 14:28:11.132515 [INFO] (runner) stopping
2020-10-12T14:28:11.132Z [INFO]  template.server: template server stopped

pod deployment
some-secret is the key of the secret I created in vault.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "app-user"
        vault.hashicorp.com/tls-skip-verify: "true"
        vault.hashicorp.com/agent-inject-secret-poc-secret: "secrets/dev/poc-secret"
        vault.hashicorp.com/agent-inject-template-secrets.env: |
          {{- with secret "secrets/dev/poc-secret" -}}
          #!/bin/bash
          export poc_secret={{ .Data.some-secret }}
          {{- end }}
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: vault-auth
      containers:
      - name: app
        image: jweissig/app:0.0.1
        args: ["sh", "-c", "source /vault/secrets/secrets.env"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  labels:
    app: app

from vault-k8s.

Hi @pksurferdad, your annotations are wrong. To override the default template, your key names need to have the same identifier.

What it should be:

vault.hashicorp.com/agent-inject-secret-poc-secrets.env: "secrets/dev/poc-secret"
vault.hashicorp.com/agent-inject-template-poc-secrets.env:
    {{- with secret "secrets/dev/poc-secret" -}}
    #!/bin/bash
        export poc_secret={{ .Data.some-secret }}
    {{- end }}

Here we're saying "for the secret we're identifying as poc-secrets.env, attach this custom template which creates an environment variable template file.

from vault-k8s.

@jasonodonnell that makes sense, but unfortunately, i'm still only seeing the poc-secret file in /vault/secrets and not the poc-secret.env file. below is the updated deployment. I assume templating is enabled by default in the vault-agent-injector? I used the latest helm chart to do the install (single/stand-alone k8s AWS EKS cluster).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "app-user"
        vault.hashicorp.com/tls-skip-verify: "true"
        vault.hashicorp.com/agent-inject-secret-poc-secret: "secrets/dev/poc-secret"
        vault.hashicorp.com/agent-inject-template-poc-secret.env: |
          {{- with secret "secrets/dev/poc-secret" -}}
          #!/bin/bash
          export poc_secret={{ .Data.some-secret }}
          {{- end }}
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: vault-auth
      containers:
      - name: app
        image: jweissig/app:0.0.1
        args: ["sh", "-c", "source /vault/secrets/poc-secret.env"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  labels:
    app: app

from vault-k8s.

@pksurferdad Your annotations are still wrong. They both need to be the same (you're missing .env). We use the identifier as the filename but they need to be same.

vault.hashicorp.com/agent-inject-secret-poc-secrets.env: "secrets/dev/poc-secret"
vault.hashicorp.com/agent-inject-template-poc-secrets.env:
    {{- with secret "secrets/dev/poc-secret" -}}
    #!/bin/bash
        export poc_secret={{ .Data.some-secret }}
    {{- end }}

• vault.hashicorp.com/agent-inject-secret-poc-secrets.env
• vault.hashicorp.com/agent-inject-template-poc-secrets.env

from vault-k8s.

yep, that was it @jasonodonnell. sorry about that. i fought a syntax error in the template for a bit (doesn't like some-secret for the key and had to change it to somesecret). below is the updated deployment. now on to figuring out why the source command on the args: parm is not firing when the container starts up.

deployment yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "app-user"
        vault.hashicorp.com/tls-skip-verify: "true"
        vault.hashicorp.com/agent-inject-secret-poc-secret.env: "secrets/dev/poc-secret"
        vault.hashicorp.com/agent-inject-template-poc-secret.env: |
          {{- with secret "secrets/dev/poc-secret" -}}
            #!/bin/bash
            export poc_secret="{{ .Data.somesecret }}"
          {{- end }}
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: vault-auth
      containers:
      - name: app
        image: jweissig/app:0.0.1
        args: ["/bin/sh", "-c", "source /vault/secrets/poc-secret.env"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  labels:
    app: app

from vault-k8s.

has anyone gotten this to work, to seed the running pod with env vars from vault? the pod starts up fine and the secrets are successfully fetched from vault, but are not available on the running pod. i can exec into the pod and manually run source /vault/secrets/poc-secret.env and sets the env var correctly.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "app-user"
        vault.hashicorp.com/tls-skip-verify: "true"
        vault.hashicorp.com/agent-inject-secret-poc-secret.env: "secrets/dev/poc-secret"
        vault.hashicorp.com/agent-inject-template-poc-secret.env: |
          {{- with secret "secrets/dev/poc-secret" -}}
            export poc_secret="{{ .Data.somesecret }}"
          {{- end }}
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: vault-auth
      containers:
      - name: app
        image: jweissig/app:0.0.1
        args: ["bin/sh", "-c", "source /vault/secrets/poc-secret.env"]

from vault-k8s.

from vault-k8s.

See #14 and https://www.vaultproject.io/docs/platform/k8s/injector/examples#environment-variable-example

from vault-k8s.

@cmbaatz Nice addition, sadly while the file is there ready to be sourced on entering the container e.g.

kubectl exec -it app-example-pod -c app /bin/sh and running env

No new env variables

However the file is there and templating applied ..

cat /vault/secrets/poc-secret 
export password="password2"
export username="user1"

from vault-k8s.

Better solution here => https://banzaicloud.com/blog/inject-secrets-into-pods-vault-revisited

from vault-k8s.

Small update about environment variables here: #14 (comment).

from vault-k8s.

@jasonodonnell i want to inject to mysql pod, MYSQL_ROOT_PASSWORD. do you have any recommend. image Mysql doesnot have source cmd, so the above solution doesnt work

from vault-k8s.

@dongnh-devops MySQL docker image supports secrets via files, so I'd suggest just using that instead.

from vault-k8s.

@jasonodonnell i know but i wanna test how vault inject env into a pod so i test with mysql pod but it havent worked yet

from vault-k8s.

has anyone gotten this to work, to seed the running pod with env vars from vault? the pod starts up fine and the secrets are successfully fetched from vault, but are not available on the running pod. i can exec into the pod and manually run source /vault/secrets/poc-secret.env and sets the env var correctly.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "app-user"
        vault.hashicorp.com/tls-skip-verify: "true"
        vault.hashicorp.com/agent-inject-secret-poc-secret.env: "secrets/dev/poc-secret"
        vault.hashicorp.com/agent-inject-template-poc-secret.env: |
          {{- with secret "secrets/dev/poc-secret" -}}
            export poc_secret="{{ .Data.somesecret }}"
          {{- end }}
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: vault-auth
      containers:
      - name: app
        image: jweissig/app:0.0.1
        args: ["bin/sh", "-c", "source /vault/secrets/poc-secret.env"]

@pksurferdad Did you get the solution for this?

from vault-k8s.

has anyone gotten this to work, to seed the running pod with env vars from vault? the pod starts up fine and the secrets are successfully fetched from vault, but are not available on the running pod. i can exec into the pod and manually run source /vault/secrets/poc-secret.env and sets the env var correctly.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "app-user"
        vault.hashicorp.com/tls-skip-verify: "true"
        vault.hashicorp.com/agent-inject-secret-poc-secret.env: "secrets/dev/poc-secret"
        vault.hashicorp.com/agent-inject-template-poc-secret.env: |
          {{- with secret "secrets/dev/poc-secret" -}}
            export poc_secret="{{ .Data.somesecret }}"
          {{- end }}
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: vault-auth
      containers:
      - name: app
        image: jweissig/app:0.0.1
        args: ["bin/sh", "-c", "source /vault/secrets/poc-secret.env"]

@pksurferdad Did you get the solution for this?

Hi @abhishekkamal96,

It does work in a way, the shell which is used to source the environment variables is also used to run the application and scripts or code referencing those environment variables will detect those environment variables (I have tested it myself). In effect when you make another shell and the environment variables are not loaded in and have to be sourced again. I do not know if this methodology is optimal for our company's use case. Let us see how good the CSI driver is.

from vault-k8s.

Using this approach:

{{- with secret "internal/data/development/api" -}}
{{- range $key, $value := .Data.data }}
export {{ $key }}={{ $value }}
{{- end }}
{{- end -}}

command: ["/bin/sh", "-c"]
args:
[ "for f in /vault/secrets/*.env; do source $f; done && npm start" ]

from vault-k8s.

Hi folks, as a few have pointed out, make sure you're setting command for your container and not just args. We've updated the injector example accordingly: https://developer.hashicorp.com/vault/docs/platform/k8s/injector/examples#environment-variable-example

from vault-k8s.

It worked for me using: command: ["/bin/bash", "-c", ". /vault/secrets/grafana.sh && . /run.sh"]

from vault-k8s.

Hi folks, it looks like we have a couple workable options here so I'll go ahead and close this.

Also note that we recently released a beta of vault-secrets-operator that will sync Vault secrets to Kubernetes secrets, which can then be used as environment variables natively: [blog] [docs]

from vault-k8s.