Comments (46)
Wrote this for myself and my team. It was pieced together using the awesome HashiCorp docs. All the examples follow what's in hashicorp/vault-guides repo and Learning modules on Hashi site. Hope one of you find this helpful.
Injecting Secrets into Kubernetes Pods via Vault
There are two distinct parts of configuring auto-injection of secrets, Kubernetes & Vault. This will go through how-to configure both in order for auto-injection to work, as well as, a few custom options that should be considered as we deploy to production. This guide combines and refines two separate tutorials provided by Learn Vault for our specific use case.
Kubernetes Auto-Injector Setup (Kubernetes)
Service Accounts
Vault Agent Injector
$ cat sa-vault-agent-injector.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-agent-injector
namespace: default
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
$ kubectl apply --filename sa-vault-agent-injector.yaml
Vault Auth
$ cat sa-vault-auth.yml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-auth
$ kubectl apply --filename sa-vault-auth.yml
Cluster Role
$ cat clusterrole-vault-agent-injector.yaml
---
# Source: vault/templates/injector-clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: vault-agent-injector-clusterrole
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
rules:
- apiGroups: ["admissionregistration.k8s.io"]
resources: ["mutatingwebhookconfigurations"]
verbs:
- "get"
- "list"
- "watch"
- "patch"
$ kubectl apply --filename clusterrole-vault-agent-injector.yaml
Cluster Role Bindings
Vault Agent Injector
$ cat clusterrolebinding-vault-agent-injector.yaml
---
# Source: vault/templates/injector-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vault-agent-injector-binding
namespace: default
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: vault-agent-injector-clusterrole
subjects:
- kind: ServiceAccount
name: vault-agent-injector
namespace: default
$ kubectl apply --filename clusterrolebinding-vault-agent-injector.yaml
Vault Auth
$ cat clusterrolebinding-vault-auth.yml
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: default
$ kubectl apply --filename clusterrolebinding-vault-auth.yml
Service
$ cat svc-vault-agent-injector.yaml
---
# Source: vault/templates/injector-service.yaml
apiVersion: v1
kind: Service
metadata:
name: vault-agent-injector-svc
namespace: default
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
spec:
ports:
- port: 443
targetPort: 8080
selector:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
component: webhook
$ kubectl apply --filename svc-vault-agent-injector.yaml
Mutating Webhook Configuration
$ cat mutatingwebhookconfigurations-vault-agent-injector.yaml
---
# Source: vault/templates/injector-mutating-webhook.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
name: vault-agent-injector-cfg
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
webhooks:
- name: vault.hashicorp.com
clientConfig:
service:
name: vault-agent-injector-svc
namespace: default
path: "/mutate"
caBundle:
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
$ kubectl apply --filename mutatingwebhookconfigurations-vault-agent-injector.yaml
As per Vault documentation, “By default, the Vault Agent Injector will process all namespaces in Kubernetes except the system namespaces kube-system and kube-public. To limit what namespaces the injector can work in a namespace selector can be defined to match labels attached to namespaces.”
namespaceSelector is the selector for restricting the webhook to only specific namespaces. This should be set to a multiline string. For more information see https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector
Deployment
$ cat deployment-vault-auto-injector.yaml
---
# Source: vault/templates/injector-deployment.yaml
# Deployment for the injector
apiVersion: apps/v1
kind: Deployment
metadata:
name: vault-agent-injector
namespace: default
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
component: webhook
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
component: webhook
template:
metadata:
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
component: webhook
spec:
serviceAccountName: "vault-agent-injector"
securityContext:
runAsNonRoot: true
runAsGroup: 1000
runAsUser: 100
containers:
- name: sidecar-injector
image: "hashicorp/vault-k8s:0.2.0"
imagePullPolicy: "IfNotPresent"
env:
- name: AGENT_INJECT_LISTEN
value: ":8080"
- name: AGENT_INJECT_LOG_LEVEL
value: info
- name: AGENT_INJECT_VAULT_ADDR
value: https://<VAULT_ADDR>:8200
- name: AGENT_INJECT_VAULT_IMAGE
value: "vault:1.3.2"
- name: AGENT_INJECT_TLS_AUTO
value: vault-agent-injector-cfg
- name: AGENT_INJECT_TLS_AUTO_HOSTS
value: vault-agent-injector-svc,vault-agent-injector-svc.default,vault-agent-injector-svc.default.svc
args:
- agent-inject
- 2>&1
livenessProbe:
httpGet:
path: /health/ready
port: 8080
scheme: HTTPS
failureThreshold: 2
initialDelaySeconds: 1
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /health/ready
port: 8080
scheme: HTTPS
failureThreshold: 2
initialDelaySeconds: 2
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 5
$ kubectl apply --filename deployment-vault-auto-injector.yaml
Note the value for AGENT_INJECT_VAULT_ADDR
this will be unique for each Vault cluster. Also take notice of AGENT_INJECT_VAULT_IMAGE
, this image must be present locally if restricting pods from only pulling images from a specific set of container repositories, such as GCR.
Kubernetes Auto-Injector Setup (Vault)
Enable and Configure Kubernetes Auth Method
$ export VAULT_SA_NAME=$(kubectl get sa vault-auth -o jsonpath="{.secrets[*]['name']}")
$ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)
$ export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
# determine Kubernetes master IP address (no https://) via `kubectl cluster-info`
$ export K8S_HOST=<K8S_MASTER_IP>
# set VAULT_TOKEN & VAULT_ADDR before next steps
$ vault auth enable kubernetes
$ vault write auth/kubernetes/config \
token_reviewer_jwt="$SA_JWT_TOKEN" \
kubernetes_host="https://$K8S_HOST:443" \
kubernetes_ca_cert="$SA_CA_CRT"
Couple things with noting.
VAULT_TOKEN
and VAULT_ADDR
environment variables must be set before trying to communicate with the Vault server.
Be aware that the vault-auth service account is associated with the authentication method. This is the service account created explicitly for the Kubernetes authentication. It is not to be associated with any other resource or action. As such, should be protected accordingly.
If access to a Vault server is restricted by firewall rules and the Kubernetes cluster is not on the same internal network as the Vault server, it is critical a path is carved from the Kubernetes cluster to the Vault server, IE. whitelisting IP address of egress traffic from cluster in Vault’s firewall rule.
Example Application Steps
1. Enable KV (v2)
$ vault secrets enable -path=internal kv-v2
2. Store secret and password
$ vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"
3. Create Vault policy
$ vault policy write internal-app - <<EOH
path "internal/data/database/config" {
capabilities = ["read"]
}
EOH
4. Create Vault authentication role
$ vault write auth/kubernetes/role/internal-app \
bound_service_account_names=internal-app \
bound_service_account_namespaces=default \
policies=internal-app \
ttl=24h
5. Create Kubernetes service account
$ cat sa-internal-app.yml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: internal-app
$ kubectl apply --filename sa-internal-app.yml
6. Create example application deployment (w/o annotations)
$ cat deployment-01-orgchart.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: orgchart
labels:
app: vault-agent-injector-demo
spec:
selector:
matchLabels:
app: vault-agent-injector-demo
replicas: 1
template:
metadata:
annotations:
labels:
app: vault-agent-injector-demo
spec:
serviceAccountName: internal-app
containers:
- name: orgchart
image: jweissig/app:0.0.1
kubectl apply --filename deployment-01-orgchart.yml
At this point the pod does not have access to the secret, as there are no Vault specific annotations within the deployment, yet.
7. Patch deployment
$ cat deployment-02-inject-secrets.yml
---
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "internal-app"
vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
$ kubectl patch deployment orgchart --patch "$(cat deployment-02-inject-secrets.yml)"
$ kubectl exec orgchart-<pod-id> --container orgchart -- cat /vault/secrets/database-config.txt
data: map[password:db-secret-password username:db-readonly-user]
metadata: map[created_time:2019-12-20T18:17:50.930264759Z deletion_time: destroyed:false version:2]
Now, there should be annotations associated with the pod. This will create the init and sidecar containers associated with the secret auto-injection. Secrets will be stored under /vault/secret. Worth noting, is the vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config" annotation. The key dictates the name of the file to be database-config.txt with the value being the path within Vault’s KV store: internal/data/database/config.
8. Templatize secret output
$ cat deployment-03-inject-secrets-as-template.yml
---
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/role: "internal-app"
vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
vault.hashicorp.com/agent-inject-template-database-config.txt: |
{{- with secret "internal/data/database/config" -}}
postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
{{- end -}}
$ kubectl patch deployment orgchart --patch "$(cat deployment-03-inject-secrets-as-template.yml)"
$ kubectl exec -it orgchart-<pod-id> -c orgchart -- cat /vault/secrets/database-config.txt
postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard
There are multiple ways to configure how the secret is formatted, using the annotation template key vault.hashicorp.com/agent-inject-template-database-config.txt. For a complete list of annotation options, see the Vault Project links in the Attribution section below.
9. Create example application deployment (w/ annotations)
$ cat deployment-04-payrole.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: payrole
labels:
app: vault-agent-injector-demo
spec:
selector:
matchLabels:
app: vault-agent-injector-demo
replicas: 1
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/role: "internal-app"
vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
vault.hashicorp.com/agent-inject-template-database-config.txt: |
{{- with secret "internal/data/database/config" -}}
postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
{{- end -}}
labels:
app: vault-agent-injector-demo
spec:
serviceAccountName: internal-app
containers:
- name: payrole
image: jweissig/app:0.0.1
$ kubectl apply --filename deployment-04-payrole.yml
Other Things Worth Noting
Secrets are bound to the service account. The service account for a deployment is specified in the spec.template.spec.serviceAccountName block. The corresponding Vault relation is specified in Vault’s authentication role
Secrets are bound to the namespace. Similar to the service account, the corresponding Vault relation is specified in Vault’s authentication role.
Errors for both service account and namespace will be manifest themselves as pods perpetually in the Init phase, never reaching ready. Errors are found within the vault-agent-init container logs, detailing the authentication error in clear English.
The token, both init and sidecar containers use to communicate with Vault, lives locally within the container at the following path: /home/vault/.token. Unsurprisingly, the token is not mounted into the primary container within the pod, making direct communications between Vault and primary container difficult.
from vault-k8s.
@smurfralf There are some dependencies in the Vault Helm project between the projects, so some stuff will need to change over there for installing the injector standalone.
I'll document the process of configuring the injector with an external Vault service after the new year since HashiCorp is currently in a holiday shutdown.
from vault-k8s.
Hi @stevegore just released in v0.2.0
. Hope that helps!
from vault-k8s.
Thanks @dcatalano-figure for the write up with the demonstrable code.
I recently delivered a HashiCorp Learn guide titled Integrate a Kubernetes Cluster with an External Vault.
In this guide, you will run Vault locally, start a Kubernetes cluster with Minikube, deploy an application that retrieves secrets from this Vault, and configure an injector only deployment to inject secrets into the pods from this Vault.
- Start Vault locally to act as the External Vault
- Start Minikube
- Determine the Vault address
- Deploy application with hard-coded Vault address
- Deploy service and endpoints to address an external Vault
- Define a Kubernetes service account
- Configure Kubernetes authentication
- Install the Vault Helm chart
from vault-k8s.
@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the method
section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_path
Hope that helps!
from vault-k8s.
Yes, I'll be adding one @buckner!
from vault-k8s.
@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the
method
section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_pathHope that helps!
Hi,
I found an annotations option that help to custom the auth_path that easier than using config map.
You can use the annotation option of vault like:
vault.hashicorp.com/auth-path: "/auth/gcp-prod-k8s-cluster/"
Config annotation like that, the vault agent will call to endpoint: https://<vault_address>:8200/v1/auth/gcp-prod-k8s-cluster/login
instead of default like that https://<vault_address>:8200/v1/auth/kubernetes/login
It's very useful when use have 1 vault cluster, and config authen with multiple k8s cluster.
Hope this help!
from vault-k8s.
@jasonodonnell thanks for the quick response.
In the light of morning I saw why I stumbled there.
Follow up question, is it possible to inject a custom auth path for the vault-agents?
from vault-k8s.
@jasonodonnell - back to the original topic of this issue...
Your comment WRT setting external URL for AGENT_INJECT_VAULT_ADDR was: "This should really be the only thing you need to do." I did it using name "vault-inj" for the helm install, and the injector is indeed functional. Yay!
But my k8s cluster also has a zombie pod vault-inj-0 showing
Readiness probe failed
because of course this vault server hasn't been initialized.
When I made a stab at setting helm value service.enabled: false
then that breaks the injector where logs show
2019-12-23T16:40:45.791Z [INFO] handler: Starting handler.. Listening on ":8080"... Updated certificate bundle received. Updating certs... 2019/12/23 16:40:46 http: TLS handshake error from 10.0.16.98:41948: no certificate available
My guess is that you have dependencies there in your automagic configuration of TLS?
At any rate, solving that isn't my point. What I asked for in the description would be nice:
... please document an example configuration that (allows configuration of an external vault server [preferably without changing the vault-helm template] and that) also disables installing vault as a service inside kubernetes.
from vault-k8s.
There's a bug in the Vault website generator causing a bunch links to break and is something we're working on fixing. https://www.vaultproject.io/docs/auth/kubernetes/ will work and sorry for the inconvenience!
from vault-k8s.
@smurfralf This scenario should work, provided pods can communicate with the external Vault cluster. You will need to set AGENT_INJECT_VAULT_ADDR
on the deployment, as you already noticed. I will document the manual installation process using the deploy
files in this repo.
from vault-k8s.
I'm struggling with using the deploy manifests from this repo, which i'm attempting to use because the helm charts are currently not working as intended with regards to global.enable variable, among other templating issues.
FWIW - I was able to get it working by changing the following
1) I added the following rbac configuration which is reused from the kubernetes auth documentation, but it's not clear to me why, as it seems the admission registration should cover it.
2) I changed AGENT_INJECT_VAULT_ADDR to my vault server url
3) I needed to make sure the vault-injector service account existed in the same namespace as my application, which makes sense in retrospect, but wasn't obvious to me in the manifests or the charts (nothing is ever obvious in a helm chart to be fair :) )
I will report back here as I become less confused :)
from vault-k8s.
Hi @ls-brentsmith,
I added the following rbac configuration which is reused from the kubernetes auth documentation, but it's not clear to me why, as it seems the admission registration should cover it.
That RBAC is not needed by the injector. The injector doesn't actually communicate with Vault at all. The rbac you listed is what Vault uses when authenticating service accounts in Kubernetes.
I changed AGENT_INJECT_VAULT_ADDR to my vault server url
This should really be the only thing you need to do.
I needed to make sure the vault-injector service account existed in the same namespace as my application, which makes sense in retrospect, but wasn't obvious to me in the manifests or the charts (nothing is ever obvious in a helm chart to be fair :) )
The deploy examples in this repository use the vault namespace. You should only need to create a namespace vault
and run the following (assuming you configured the AGENT_INJECT_VAULT_ADDR
with your external Vault server:
kubectl create namespace vault
kubectl create -f ./deploy/injector-rbac.yaml --namespace=vault
kubectl create -f ./deploy/injector-service.yaml --namespace=vault
kubectl create -f ./deploy/injector-mutating-webhook.yaml --namespace=vault
kubectl create -f ./deploy/deployment.yaml --namespace=vault
kubectl get pods --namespace=vault
from vault-k8s.
Thank you for the quick response!
I was just about to come comment the same thing.
from vault-k8s.
Are there plans to add an annotation for custom auth paths? It'd be nice to have a choice between a configmap or just annotations.
from vault-k8s.
@jasonodonnell I have a setup of external vault server with consul backend and placed a load balancer in front of vault server in an EKS environment. Now m app has been deployed in separate EKS cluster where I mentioned AGENT_INJECT_VAULT_ADDR as the above mentioned ELB(Port 80). Now my app pod becomes not ready after I injected the init container . STATUS becomes 'Init:0/1' . I think it is something related to token initialization. How can I mention that token here and retrieve the secrets inside the Pod?
from vault-k8s.
@VinothChinnadurai Did you check the init container logs to see what's happening?
kubectl logs <name of pod> -c vault-agent-init
from vault-k8s.
Yes @jasonodonnell . Please find it here
`URL: PUT http://vault.demo.svc:8200/v1/auth/kubernetes/login
Code: 400. Errors:
- invalid role name "exampleapp-role"" backoff=1.580907647
2019-12-24T10:09:31.660Z [INFO] auth.handler: authenticating
2019-12-24T10:09:31.667Z [ERROR] auth.handler: error authenticating: error="Error making API request`
Not sure why it still hits some other service(vault.demo.svc) instead of ELB address
from vault-k8s.
@VinothChinnadurai Double check the env is set correctly because that looks like the default
vault-k8s/deploy/injector-deployment.yaml
Line 36 in 611492d
Next retrigger the injection by patching the pod. You'll need to set the status annotation to vault.hashicorp.com/agent-inject-status: update
to retrigger the injection (the config needs to be refreshed on the agent).
from vault-k8s.
@jasonodonnell Happy New year! Thanks. It worked for me! But I find an issue with the syncing part where I have updated my secrets through vault API command
curl \ --header "X-Vault-Token: xxx" \ --request POST \ --data @payload.json \ http://myelbaddress:8200/v1/secret/helloworld
Payload.json has
{ "data": { "foo": "bar", "zip": "zap" } }
Post then in the GET command of API, I can receive the updated secrets.
I tried to see the same by logging inside my app pod and verified the file /vault/secret/helloworld
Where it still showed the older values.
Kindly help me with what might be the cause.
from vault-k8s.
@VinothChinnadurai Lower the TTL of your secret. Updates are based on TTL of the secret.
from vault-k8s.
@jasonodonnell does this also work with istio service mesh?
from vault-k8s.
@trx35479 We haven't done explicit testing with istio yet, so we can not verify if this will or will not work with it. If you happen to test this, please let us know what you find.
from vault-k8s.
@VinothChinnadurai Double check the env is set correctly because that looks like the default
vault-k8s/deploy/injector-deployment.yaml
Line 36 in 611492d
Next retrigger the injection by patching the pod. You'll need to set the status annotation to
vault.hashicorp.com/agent-inject-status: update
to retrigger the injection (the config needs to be refreshed on the agent).
==========================================================
@jasonodonnell
I have consul+vault installed on one Kubernetes cluster. On the other cluster, I am trying to deploy vault-k8s injector for webhook - https://github.com/hashicorp/vault-k8s.git
Init pod returns the following errors. how can it be fixed? Vault address has been changed to http://vlt.consulvault.172.31.101.63.xip.io
Both the clusters are in the same network and curl command returns the response.
I think I may have to pass the root token(or register secret with Vault) in order to authenticate.
Thank you
curl \
> -H "X-Vault-Token: token" \
> -X GET \
> http://vlt.consulvault.172.31.101.63.xip.io/v1/secret/helloworld
{"request_id":"***","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"password":"foobarbazpass","username":"foobaruser"},"wrap_info":null,"warnings":null,"auth":null}
==> Vault server started! Log data will stream in below:
2020-01-28T18:27:10.042Z [INFO] sink.file: creating file sink
2020-01-28T18:27:10.042Z [INFO] sink.file: file sink configured: path=/home/vault/.token mode=-rw-r-----
==> Vault agent configuration:
Cgo: disabled
Log Level: info
Version: Vault v1.3.1
2020-01-28T18:27:10.042Z [INFO] auth.handler: starting auth handler
2020-01-28T18:27:10.042Z [INFO] auth.handler: authenticating
2020-01-28T18:27:10.042Z [INFO] template.server: starting template server
2020/01/28 18:27:10.042917 [INFO] (runner) creating new runner (dry: false, once: false)
2020/01/28 18:27:10.043280 [INFO] (runner) creating watcher
2020-01-28T18:27:10.043Z [INFO] sink.server: starting sink server
2020-01-28T18:27:10.275Z [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login
Code: 500. Errors:
* lookup failed: [invalid bearer token, square/go-jose: error in cryptographic primitive]" backoff=2.382848811
2020-01-28T20:21:34.792Z [INFO] auth.handler: authenticating
2020-01-28T20:21:34.806Z [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login
Code: 500. Errors:
* lookup failed: [invalid bearer token, square/go-jose: error in cryptographic primitive]" backoff=1.805414534
2020-01-28T20:21:36.612Z [INFO] auth.handler: authenticating
2020-01-28T20:21:36.758Z [ERROR] auth.handler: error authenticating: error="Put http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login: dial tcp: lookup vlt.consulvault.172.31.101.63.xip.io on 10.43.0.10:53: no such host" backoff=1.998397118
2020-01-28T20:21:38.757Z [INFO] auth.handler: authenticating
2020-01-28T20:21:38.761Z [ERROR] auth.handler: error authenticating: error="Put http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login: dial tcp: lookup vlt.consulvault.172.31.101.63.xip.io on 10.43.0.10:53: no such host" backoff=2.780268301
2020-01-28T20:21:41.541Z [INFO] auth.handler: authenticating
from vault-k8s.
@jasonodonnell @shlee322 I've noticed the code for setting a custom path for the Kubernetes Auth method has been committed to master. Any chance you guys are going to do a release of that soon?
from vault-k8s.
@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the
method
section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_path
Hope that helps!Hi,
I found an annotations option that help to custom the auth_path that easier than using config map.
You can use the annotation option of vault like:
vault.hashicorp.com/auth-path: "/auth/gcp-prod-k8s-cluster/"
Config annotation like that, the vault agent will call to endpoint:
https://<vault_address>:8200/v1/auth/gcp-prod-k8s-cluster/login
instead of default like thathttps://<vault_address>:8200/v1/auth/kubernetes/login
It's very useful when use have 1 vault cluster, and config authen with multiple k8s cluster.
Hope this help!
@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the
method
section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_path
Hope that helps!Hi,
I found an annotations option that help to custom the auth_path that easier than using config map.
You can use the annotation option of vault like:
vault.hashicorp.com/auth-path: "/auth/gcp-prod-k8s-cluster/"
Config annotation like that, the vault agent will call to endpoint:
https://<vault_address>:8200/v1/auth/gcp-prod-k8s-cluster/login
instead of default like thathttps://<vault_address>:8200/v1/auth/kubernetes/login
It's very useful when use have 1 vault cluster, and config authen with multiple k8s cluster.
Hope this help!
Indeed, this is a new feature in 0.2.0
from vault-k8s.
I see this issue is resolved but could someone guide me to the final documentation. We have an external vault gke cluster and multiple application gke clusters. I would like to see how we can use this tool to integrate our existing vault to inject secrets to all those clusters.
from vault-k8s.
@sandeepkatthi99: https://learn.hashicorp.com/vault/getting-started-k8s/external-vault
from vault-k8s.
@jasonodonnell , as per this documentation it is using a minikube and it is using http protocol to connect to the external vault sever. Mine is an external vault server with https protocol which uses a ca authorized certificate. in this case how to change the helm command to create injector pod with certificate options? Please help me. If i just replace http with https I am gtting below error.
auth.handler: authenticating
2020-03-26T10:03:01.452Z [ERROR] auth.handler: error authenticating: error="Put https://****/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=2.96206
I have built my vault cluster using https://codelabs.developers.google.com/codelabs/vault-on-gke/index.html?index=..%2F..cloud#18
from vault-k8s.
Hi @sandeepkatthi99,
You need to create a Kube secret with the servers CA certificate (in the namespace with your app), then using annotations configure the Vault Agent to use it:
vault.hashicorp.com/tls-secret: "<NAME OF KUBE SECRET>"
vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"
from vault-k8s.
This issue should be closed since the original description has been satisfied by vault-helm v0.4 with injector.externalVaultAddr
value.
from vault-k8s.
Thanaks @jasonodonnell jasonodonnell I will try the same.
from vault-k8s.
@burtlo Thanks for the learning guide.
I just wanted to let you know that in the Configure Kubernetes authentication section, the Kubernetes authentication
link pointing to https://www.vaultproject.io/docs/auth/kubernetes.html displays a "Page Not Found".
from vault-k8s.
@dcatalano-figure thanks for sharing your setup! I have a question though - I'm trying to go through with your setup however we have multiple clusters that would like to access vault secret. How do we go about that?
from vault-k8s.
@therealsamlin we're mounting the Kubernetes auth method at a different path for each cluster. Then set vault.hashicorp.com/auth-path
appropriately on the injector per cluster.
from vault-k8s.
@dcatalano-figure thanks for sharing your setup! I have a question though - I'm trying to go through with your setup however we have multiple clusters that would like to access vault secret. How do we go about that?
basically what @stevegore said. would like to add that auth path value must be prefixed with auth/
in where if the mount was named k8s-cluster-01 the value would be auth/k8s-cluster-01
from vault-k8s.
In my case, I have vault server running on GKE cluster A and I have installed Vault Agent Injector service on cluster B by passing externalVaultAddr which is exposed over http.
But after deploying sample application on B along with necessary annotations to inject vault secrets, I am getting below error in vault-agent-init container of my sample app
2020-08-26T08:24:59.554Z [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT http://35.225.127.203/v1/auth/kubernetes/login
Code: 403. Errors:
* permission denied" backoff=2.908150444
2020-08-26T08:25:02.463Z [INFO] auth.handler: authenticating
@jasonodonnell Any idea how to resolve this?
from vault-k8s.
Hi, I am also facing the same issue. (I have 2 gke cluster setup , Cluster A vault is installed with internal LB, cluster B for application
Error writing data to auth/kubernetes/login: Error making API request.
URL: PUT http://gke_internal_LB_IP:8200/v1/auth/kubernetes/login
Code: 403. Errors:
- permission denied
and logs
[ERROR] auth.kubernetes.auth_kubernetes_c78cbc33: login unauthorized due to: Post "https://cluster_app_ip/apis/authentication.k8s.io/v1/tokenreviews": dial tcp cluster_app_ip:443: i/o timeout
from vault-k8s.
Hello,
This question may have been answered earlier - not sure.
While performing the setup on EKS side , after executing all the 4 yaml files under deploy folder, i am able to create the agent injector POD.
However, when trying to perform the setup on the vault side, i am trying to obtain the JWT token as below , it complains of the missing secret token in secrets :
export SERVICEACCOUNT_JWT_TOKEN=$(kubectl get secret -o jsonpath="{.data.token}" $(kubectl get serviceaccount vault-injector -n vault -o jsonpath="{.secrets[0].name}") | base64 -d)
Error from server (NotFound): secrets "vault-injector-token-p5jjp" not found
I verify that the secret does in exist with kubectl get secrets -n vault command
Does the secret need to be created manually ? Is there a yaml to execute while deploying the agent injector itself ?
Please let me know.
from vault-k8s.
Hi @sandeepkatthi99,
You need to create a Kube secret with the servers CA certificate (in the namespace with your app), then using annotations configure the Vault Agent to use it:
vault.hashicorp.com/tls-secret: "<NAME OF KUBE SECRET>" vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"
Hello, @jasonodonnell
please explain how to do it, I am new in Vault
from vault-k8s.
@asl-cloud99 We are also having the similar use case (I have 2 gke cluster setup with same VPC, Cluster A vault is installed with internal LB, cluster B for application). Are you able to make it to work? If so, can you please share the steps?
from vault-k8s.
I deployed the injector via the standard helm chart, with --set injector.externalVaultAddr
as describe in the Integrate a Kubernetes Cluster with an External Vault tutorial, and I'm seeing a TLS handshake error in the injector log whenever I deploy an annotated POD (the devwebapp-with-annotations from the tutorial).
[centos@ip-10-5-58-194 (vaulttest) ~]$ kubectl logs vault-agent-injector-5657ff569b-kdn8n -c sidecar-injector
2021-06-15T19:35:24.646Z [INFO] handler: Starting handler..
Listening on ":8080"...
2021-06-15T19:35:24.654Z [INFO] handler.auto-tls: Generated CA
2021-06-15T19:35:24.655Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-16T17:11:24.000Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-17T14:47:24.000Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-18T12:23:24.000Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-18T15:11:36.470Z [ERROR] handler: http: TLS handshake error from 127.0.0.1:56344: EOF
I've plenty of other TLS handshake issues reported with a bad certificate error but not this EOF variant.
from vault-k8s.
A colleague "changed service mesh mtls strict policy to permissive" and that got rid of the TLS handshake error but the injector still wasn't injecting the init/agent containers. Then she disabled the istio sidecar from the vault-agent-injector POD, and it started working.
from vault-k8s.
Hi Guys, I've tried all things shared here and still getting:
2021-06-20T21:38:24.892Z [INFO] auth.handler: authenticating
2021-06-20T21:39:24.893Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1.87s
HELM:
helm install vault hashicorp/vault --set
"server.dev.enabled=true,server.extraEnvironmentVars.VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200,server.extraEnvironmentVars.VAULT_ADDR=http://vault:8200"
APP.YML:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vault-app
spec:
selector:
matchLabels:
app: vault-app
template:
metadata:
labels:
app: vault-app
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-init-first: "true"
vault.hashicorp.com/agent-inject-secret-hello.txt: secret/hello
vault.hashicorp.com/role: vault-app
vault.hashicorp.com/agent-pre-populate: "false"
vault.hashicorp.com/service: "http://vault:8200"
spec:
serviceAccountName: vault-app
containers:
- name: debian
image: debian:latest
command: [sleep, infinity]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-app
namespace: default
labels:
app: vault-app
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: vault-app
namespace: default
labels:
app: vault-app
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
namespace: default
name: vault-app
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-ingress
spec:
podSelector: {}
ingress:
- {}
policyTypes:
- Ingress
VAULT_0 conf:
vault kv put secret/hello foo=world
vault auth enable kubernetes
vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
# disable_local_ca_jwt=true \
# disable_iss_validation=true
vault policy write vault-app - <<'EOF'
path "secret/hello" {
capabilities = ["create", "read", "update", "delete", "list"]
}
EOF
vault write auth/kubernetes/role/vault-app \
bound_service_account_names=vault-app \
bound_service_account_namespaces=* \
policies=vault-app ttl=24h
Would you please help me to spot what is wrong here? I'd really really appreciate some help ...
P.D: I've check all the rolebindings + services accounts + webmutations elated to vault and all of them seem to be correct ...
from vault-k8s.
Hi,
I followed https://learn.hashicorp.com/tutorials/vault/kubernetes-external-vault?in=vault/kubernetes#install-the-vault-helm-chart-configured-to-address-an-external-vault this document.
Installed Vault on one cluster and trying to retrieve secrets into the pod which is deployed on another cluster.
However, I am facing authentication issue as shown below
2021-12-22T11:55:02.769Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT http://51.143.61.191:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=4m1.09s
I have enabled debug, here is the output of it
2021-12-22T11:08:10.887Z [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"{{ with secret "secret/data/devwebapp/config" }}{{ range $k, $v := .Data }}{{ $k }}: {{ $v }}\n{{ end }}{{ end }}","CreateDestDirs":true,"Destination":"/vault/secrets/credentials.txt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://51.143.61.191:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"DefaultLeaseDuration":300000000000},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}
Could someone please help in resolving this issue...
from vault-k8s.
Hi, I followed https://learn.hashicorp.com/tutorials/vault/kubernetes-external-vault?in=vault/kubernetes#install-the-vault-helm-chart-configured-to-address-an-external-vault this document. Installed Vault on one cluster and trying to retrieve secrets into the pod which is deployed on another cluster.
However, I am facing authentication issue as shown below
2021-12-22T11:55:02.769Z [ERROR] auth.handler: error authenticating: error= | Error making API request. | | URL: PUT http://51.143.61.191:8200/v1/auth/kubernetes/login | Code: 403. Errors: | | * permission denied backoff=4m1.09s
I have enabled debug, here is the output of it
2021-12-22T11:08:10.887Z [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"{{ with secret "secret/data/devwebapp/config" }}{{ range $k, $v := .Data }}{{ $k }}: {{ $v }}\n{{ end }}{{ end }}","CreateDestDirs":true,"Destination":"/vault/secrets/credentials.txt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://51.143.61.191:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"DefaultLeaseDuration":300000000000},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}
Could someone please help in resolving this issue...
Suggest you check the auth Method of the Cluster from where you are trying to retrieve secret.
from vault-k8s.
Related Issues (20)
- Injector failure mode prevents Pod deletion HOT 3
- Injector sidecar is working for inject Pod manifest but Deployment manifest doesn't work HOT 1
- Agent injector should set a maxSize for its tmpfs mount
- Vault agent overwrites kubernetes managedFields
- Allow configuration of the init/sidecar container names globally HOT 1
- Injected config tries to use IRSA token instead of the k8s service account token
- Webhook tries to add initContainer during UPDATE HOT 4
- Stuned deleting of a pod whose parents are job.
- vault.hashicorp.com/agent-init-first does not work with init containers coming from annotations
- Azure authentication method doesn't work with federated token
- Support for an agent-image built FROM scratch
- Auth config block can support common arguments from env and flags
- Tokens not revoked on Vault Agent Shutdown created via a Job using the /agent/v1/quit endpoint HOT 3
- Pipeline Request: Rebuild Dockerhub Image HOT 1
- Support for a securityContext.seccompProfile configuration HOT 1
- Support vault secret inject while the main pod "automountServiceAccountToken" set false HOT 1
- [controller-runtime] log.SetLogger(...) was never called; logs will not be displayed. HOT 1
- Sidecar agent does not handle manually rotated static database secret
- Inject the Agent as a native sidecar HOT 2
- Allow patching the Agent's configuration HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vault-k8s.