doitintl / kube-secrets-init Goto Github PK
View Code? Open in Web Editor NEWKubernetes mutating webhook for `secrets-init` injection
License: Apache License 2.0
Kubernetes mutating webhook for `secrets-init` injection
License: Apache License 2.0
Hi,
I work at Oracle and a customer is using this component, I clone your repo and make an update to support Oracle Cloud Vault
please, check it out and when possible include this directly on your repo.
I don't know Go Lang, so I don't create the files to test the solution and I comment a lot of lint that was on Makefile just to build the container,
The solution is working to Oracle Cloud Vault, but don't have automated test files.
Checkout on my git
https://github.com/herberthnilsen/secrets-init
https://github.com/herberthnilsen/kube-secrets-init
I'm getting the following error after deploying my applications that are calling a secret from Google Secret Manager via secrets-init.
Internal error occurred: failed calling webhook "secrets-init.doit-intl.com": expected webhook response of admission.k8s.io/v1, Kind=AdmissionReview, got /, Kind=
Additionally, I had to add admissionReviewVersions
and sideEffects
to the mutatingwebhook-bundle.yaml
file otherwise, the webhook wouldn't have deployed succesfully. These were omitted from your documentation,however.
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: mutating-secrets-init-webhook-cfg
labels:
app: secrets-init-webhook
webhooks:
- name: secrets-init.doit-intl.com
clientConfig:
service:
name: secrets-init-webhook-svc
namespace: default
path: "/pods"
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURLekNDQWhPZ0F3SUJBZ0lSQVAyQ3BnQjlEVGpZbk5xSVBlM01aTTB3RFFZSktvWklodmNOQVFFTEJRQXcKTHpFdE1Dc0dBMVVFQXhNa09ETTVPVFptTW1ZdE1qSmtPQzAwT0RaakxUazNaVGt0TXpsbE0yWXlObUV5T0RaagpNQjRYRFRJeE1ETXhNREF5TWpZMU0xb1hEVEkyTURNd09UQXpNalkxTTFvd0x6RXRNQ3NHQTFVRUF4TWtPRE01Ck9UWm1NbVl0TWpKa09DMDBPRFpqTFRrM1pUa3RNemxsTTJZeU5tRXlPRFpqTUlJQklqQU5CZ2txaGtpRzl3MEIKQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeTVRU2ZDTitoWERoTG04ZGRrN2Zzbk1HMG84bm9ZeVhUaC9ZeW1UZApiUGRrRGFTT3g1eU9weWtGb1FHV3RNaWR6RTNzd2FWd0x6WjFrdkpCaHZIWm43YzBsRDBNKytJbmNGV2dqZjEzCjdUS2RYZjI1TEFDNkszUVl3MmlzMTc5L1U1U2p2TUVCUFdzMkpVcFZ1L2s2Vm50UmZkMWtLSmpST2tVVTVMWlgKajVEZncyb2prNlZTeSs3MDh4aTBlZU14bjNzUzU1Q3hUSGJzNkdBWTRlOXVRUVhpT2dCWXl4UG90Nlk2Vk9WSApDcW1yTXQ3V1ZTQ0xvOVJDb1V3NjlLSnQ5aWVDdW13QnpWMW4xNXF5bExhNXF0MWdWa3h2RkF3MDRweUxWMnBCCmowSFNXdVQ3L2w4Q1dYeXhMdnlNNFoxeEc3VFQva3FSMElHRyt5YWI4Snk3cFFJREFRQUJvMEl3UURBT0JnTlYKSFE4QkFmOEVCQU1DQWdRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVVRFSjFJN3phSGJkRQp0amxTUEtJdGU2VlhXbTB3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQU1YaU9BbGcweDlqTE5Zbm1IS3MrNW1ECmVhbnhUdmZxWHFCSlphK1ZPZnFYNm4xMzBncGk2cnM2K2lxUnh6bkVtVUJVZWJGM2s4c2VSUFBzenFvRzh4OFMKOXNHaE1idlBkVjZleTByMUR0VGFQQnk2blFXUUdqTzhXV2JNd21uakRlODhCQzZzckp1UEpCM0ZLVVYwSWF1NQo5bHhKaW5OamtoMXp4OHpYNVV2anNXeXg2dk5Ha0lZQy9KR2pTem5NRGhzZEVEbmE0R2FqOHR0TUlPWjduRG9JCkdkeWxCNnFwVUgvZmVsMURoTGlRWFIvL0cyRkdHRERQc3BUc1ZmczV5N2p3d2NURGgwYnpSZmpjdjhBRXR1cDQKWjlQSU9hNUdhN0NXbVJIY0FHSXBnSGdzUTZ5VC95N3krVVluM1pmVW44NEYwdERFMi9HbnN5ekRWZDM4cHZBPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
admissionReviewVersions: ["v1"]
sideEffects: None
rules:
- operations: [ "CREATE" ]
apiGroups: ["*"]
apiVersions: ["*"]
resources: ["pods"]
I noticed the arguments passed to secrets-init are hard coded to just --project
. However, secrets-init has other arguments that it accepts. It would be nice if we could specify exactly what arguments are set by the webhook controller.
Hi,
This is a great solution to deal with k8s secrets. Is this repository still maintained?
Thank you
Shaked
I'm trying to use the webhook on google cloud, but I can't seem to be able to authenticate against GCR. There was an issue about it: #1, but it was closed without a resolution.
GCR uses service account keys for authentication, which is a JSON payload and it doesn't seem to work with neither password nor "auth" (using auth field from the docker config) authentication.
Am I missing something? Is there a solution already? Thanks!
Currently tests are not executed by CI.
Could you please add support for ECR based repos?
If I got this correct, kube-secrets-init currently only supports public image repos. Even with right read permissions on pod level, ECR requires login.
I would love to use this tool as it is, but that's my blocker atm.
thanks in advance.
Hi everyone!
I found your project and I have some questions about the functionality of this, when the k8s cluster has dinamic namespaces.
We use gitlab for CI/CD and create one namespace for every branch so, I want apply your config but I don´t identify what is the component that I need add to the deploy process for every service to use the inyector.
I use helm to make the deploy, and this deploy has a service account.
I understood that if I install kube-secrets-init to every namespace this can work, but I think that is a bad way to do this task.
Sorry for my english, I tried to explain my doubt.
If you can explain more about the relationship between the inyector and the common services of every microservices arquitecture that I want that use this inyector, I really appreciate.
Greetings!
Hi.
It seems it doesn't work with private registry.
I've received error below:
time="2020-04-11T17:36:35Z" level=warning msg="no tracer active"
time="2020-04-11T17:36:35Z" level=info msg="listening on https://:8443"
time="2020-04-11T17:38:45Z" level=debug msg="reviewing request c8cce15f-2e93-4c17-95d4-5bcc23ab6c9f, named: dev/"
time="2020-04-11T17:38:45Z" level=debug msg="no pod init containers were mutated"
2020/04/11 17:38:45 registry.ping url=https://eu.gcr.io/v2/
time="2020-04-11T17:38:45Z" level=error msg="admission webhook error: cannot create client for registry: Get \"https://eu.gcr.io/v2/\": http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"Unauthorized access.\\\"}]}\")"
Hi,
When I look into the code I see that it should be possible to use the "placeholder" for SecretKeyRef but when I use it I'm getting the validation error
is invalid: spec.template.spec.containers[0].env[6].valueFrom.secretKeyRef.key: Invalid value: a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name', or 'KEY_NAME', or 'key-name', regex used for validation is '[-. _a-zA-Z0-9]+')
0.5.0
0.5.0
0.9.3
get secrets
- due to security requirements, we denied this permissionfailurePolicy: Fail
debug
google
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: svcc
name: svcc
namespace: secretspoc
spec:
replicas: 1
selector:
matchLabels:
app: svcc
strategy: {}
template:
metadata:
labels:
app: svcc
spec:
containers:
- image: alpine:3.18.4
name: second
command:
- /bin/sh
- -c
- |
while true
do
date
env | grep $FILTER
echo "====="
sleep $SLEEP_DURATION
done
resources:
requests:
cpu: "150m"
memory: "150Mi"
limits:
cpu: "150m"
memory: "150Mi"
env:
- name: EXIT_EARLY
value: "true"
- name: FILTER
value: "GIT"
- name: SLEEP_DURATION
value: "10"
- name: GIT_PASSWORD
value: gcp:secretmanager:<password-secret>/versions/latest
- name: GIT_SSH_KEY
valueFrom:
secretKeyRef:
name: svcc-secrets
key: GIT_SSH_KEY
optional: true
envFrom:
- configMapRef: # GIT_USERNAME
name: svcc-configmap
optional: true
- image: alpine:3.18.4
imagePullPolicy: Always
command:
- /bin/sh
- -c
- |
while true
do
date
env | grep $FILTER
echo "====="
sleep $SLEEP_DURATION
done
name: first
resources:
requests:
cpu: "150m"
memory: "150Mi"
limits:
cpu: "150m"
memory: "150Mi"
env:
- name: EXIT_EARLY
value: "true"
- name: FILTER
value: "GIT"
- name: SLEEP_DURATION
value: "10"
- name: GIT_PASSWORD
value: gcp:secretmanager:<password-secret>/versions/latest
- name: GIT_USERNAME
valueFrom:
configMapKeyRef:
name: svcc-configmap
key: GIT_USERNAME
envFrom:
- secretRef: # ssh-key
name: svcc-secrets
optional: true
serviceAccountName: <sa-name> #sa had permission to read secrets from GCP SM
status: {}
---
apiVersion: v1
data:
GIT_SSH_KEY: ZHVtbXkK # dummy secret, not a reference to GCP SM
kind: Secret
metadata:
name: svcc-secrets
namespace: secretspoc
---
apiVersion: v1
data:
GIT_USERNAME: gcp:secretmanager:<name>/versions/latest
kind: ConfigMap
metadata:
name: svcc-configmap
namespace: secretspoc
Because kube-secret-init
cannot read any kubernetes secrets (as configured in the RBAC) and, the pod mutation is blocked regardless of:
optional: true
set in the pod manifest (for both envFrom
and env:secretKeyRef
)env:value
and configmaps
) referencing GCP SM secretsPlus, the logs of the kube-secrets-init
are not specific about any of this. The only log line is: level=debug msg="no pod init containers were mutated"
.
Only the ReplicaSet controller shows some information:
Warning FailedCreate 66s replicaset-controller Error creating: Internal error occurred: failed calling webhook "pods.kube-secrets-init.admission.doit-intl.com": failed to call webhook: an error on the server ("{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1beta1\",\"response\":{\"uid\":\"6ebe4fad-6f30-49d1-8bf0-b2880ff77b8a\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"could not mutate object: failed to mutate pod: : failed to mutate containers for pod : failed to look for envFrom: failed to get secret secretspoc/svcc-secrets: failed to get secret secretspoc/svcc-secrets: secrets \\\"svcc-secrets\\\" is forbidden: User \\\"system:serviceaccount:secretspoc:kube-secrets-init\\\" cannot get resource \\\"secrets\\\" in API group \\\"\\\" in the namespace \\\"secretspoc\\\"\"}}}") has prevented the request from succeeding
Warning FailedCreate 30s (x5 over 66s) replicaset-controller (combined from similar events): Error creating: Internal error occurred: failed calling webhook "pods.kube-secrets-init.admission.doit-intl.com": failed to call webhook: an error on the server ("{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1beta1\",\"response\":{\"uid\":\"e5d08f29-bd31-40ca-9c58-d2c027f5472f\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"could not mutate object: failed to mutate pod: : failed to mutate containers for pod : failed to look for envFrom: failed to get secret secretspoc/svcc-secrets: failed to get secret secretspoc/svcc-secrets: secrets \\\"svcc-secrets\\\" is forbidden: User \\\"system:serviceaccount:secretspoc:kube-secrets-init\\\" cannot get resource \\\"secrets\\\" in API group \\\"\\\" in the namespace \\\"secretspoc\\\"\"}}}") has prevented the request from succeeding
kube-secrets-init
.Hi,
I'm getting
Error: execution error at (kube-secrets-init/templates/deployment.yaml:46:28): Secret provider is required!
during execution of
helm install --generate-name --wait skm/kube-secrets-init
Did I miss something ?
As of November, Docker Hub will enforce radical rate limiting policies which affects basically the whole world.
As a response to that change, GitHub introduced a new image registry.
Many projects are migrating away from Docker Hub, because the 100 pulls/IP/6 hours is hitting Kubernetes users hard (especially with private clusters).
It would be nice to have images pushed to ghcr.io as well. I'd be happy to provide a PR to your GitHub Actions workflows.
https://doitintl.zendesk.com/agent/tickets/119168
^-- non-public, internal link
A customer encountered a failure scenario where:
Break Fix Patch Solution:
Potential Permanent solution:
Investigate implementing or documenting namespace whitelisting / blacklisting.
I recall OPA GK (open policy agent gatekeeper) is another tool that uses webhooks that can have a similar failure scenario, as it's a generic issue across Kubernetes. (improvement is in upstream development's feature pipeline https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) That tool solved this by implementing namespace whitelisting / blacklisting options.
If the webhook would never run against kube-system namespace, it should decouple kubedns / make it so kubedns no longer requires the webhook to work.
I'm getting this error and curious if anybody else has experienced this. Seems like the kube-secret-init binary doesn't like what it's getting back from the SSM endpoint?
time="2020-10-07T19:00:18Z" level=error msg="failed to resolve secrets" error="failed to get secret from AWS Parameters Store: RequestError: send request failed\ncaused by: Post \"https://ssm.us-east-1.amazonaws.com/\": x509: certificate signed by unknown authority"
When the pod manifest defines an environment variable using this format, the mutating webhook fails.
envFrom:
- secretRef:
name: <secret-name>
optional: true
or
env:
valueFrom:
secretKeyRef:
name: <secret-name>
key: <key>
optional: true
So, normally, if the secret is not found, kubernetes starts the pod as the secret is marked as optional
. However, the kube-secrets-init
mutating webhook is failing the mutating request, blocking pod scheduling (which normally should work).
In theory, kube-secrets-init should not break the normal kubernetes functionality.
Currently it is fails if image for container is from private aws ecr registry:
2021/07/13 15:18:36 registry.ping url=https://857306286857.dkr.ecr.eu-west-1.amazonaws.com/v2/
FYI I opened an issue in the upstream webhook repo to add support for this: slok/kubewebhook#72
Hello team,
There is an issue when using kube-secrets-init
as an admission webhook with the latest version. The initContainer injected in the pod fails with the following error:
Error: failed to start container "copy-secrets-init": Error response from daemon:
OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init
caused \"rootfs_linux.go:58:
mounting \\\"/var/lib/kubelet/pods/f3148abf-de2a-4ebd-8288-
90b3afb7f708/volumes/kubernetes.io~empty-dir/secrets-init-bin\\\" to rootfs
\\\"/var/lib/docker/overlay2/d4827f88cc99549275789bcaf477f905fbc7e24847d1ae110e680fb2e759c064/merged\\\" at
\\\"/var/lib/docker/overlay2/d4827f88cc99549275789bcaf477f905fbc7e24847d1ae110e680fb2e759c064/merged/secrets-
init/bin\\\" caused \\\"not a directory\\\"\"": unknown:
Are you trying to mount a directory onto a file (or vice-versa)?
Check if the specified host path exists and is the expected type.
Kind regards,
Sergei
Hello,
In your example folder you have top-secret with a long encrypted - how did you create that secret? Where did that string come from? Is that generated from the ARN somehow or is it just made up?
David
Hi! I've enabled workload identity and link GSA and K8S_SA of services. But receive this error:
time=“2020-04-16T21:22:10Z” level=error msg=“failed to resolve secrets” error=“failed to get secret from Google Secret Manager: rpc error: code = Unauthenticated desc = transport: Get \“http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform\“: net/http: timeout awaiting response headers”
Got the following error trying to create the mutatingwebhookconfiguration:
% kubectl create -f deployment/mutatingwebhook-bundle.yaml
error: error validating "deployment/mutatingwebhook-bundle.yaml": error validating data: [ValidationError(MutatingWebhookConfiguration.webhooks[0]): missing required field "sideEffects" in io.k8s.api.admissionregistration.v1.MutatingWebhook, ValidationError(MutatingWebhookConfiguration.webhooks[0]): missing required field "admissionReviewVersions" in io.k8s.api.admissionregistration.v1.MutatingWebhook]; if you choose to ignore these errors, turn validation off with --validate=false
Seems like some more fields are needed in the yaml.
Hi
currently it is fails if container image is stored on the private gcr registry with the following error:
Error creating: Internal error occurred: failed calling webhook "pods.kube-secrets-init.admission.doit-intl.com": an error on the server ("{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1beta1\",\"response\":{\"uid\":\"cf21320d-1d33-4fc6-a6f5-65a550ed25ef\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"could not mutate object: failed to mutate pod: : cannot create client for registry: Get \\\"https://gcr.io/v2/\\\": http: non-successful response (status=401 body=\\\"{\\\\\\\"errors\\\\\\\":[{\\\\\\\"code\\\\\\\":\\\\\\\"UNAUTHORIZED\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"Unauthorized access.\\\\\\\"}]}\\\")\"}}}") has prevented the request from succeeding
it is impossible to solve this by providing registry credentials - as there is no static password, it should support gcr method of authentication
Setup:
This setup works on all but two deployments. The only notable difference between pods/rs working vs non working pods/rs if there is multiple defined containers. To note running 0.4.2/0.4.0 works. From looking at the changes in 0.4.3 it appears logic around registry secrets are handled has changed.
Warning FailedCreate 17m replicaset-controller Error creating: Internal error occurred: failed calling webhook "pods.kube-secrets-init.admission.doit-intl.com": an error on the server ("{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1beta1\",\"response\":{\"uid\":\"88fceb5d-b4a1-437b-b602-b578bf037b07\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"could not mutate object: failed to mutate pod: : cannot fetch image descriptor: GET https://us-docker.pkg.dev/v2/token?scope=repository%3A<project>%2F<repository>%2F<image>%3Apull\\u0026service=us-docker.pkg.dev: DENIED: Permission \\\"artifactregistry.repositories.downloadArtifacts\\\" denied on resource \\\"projects/<project>/locations/us/repositories/<repository>\\\" (or it may not exist)\"}}}") has prevented the request from succeeding
according to the documentation
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#certificatesigningrequest-v122
the certificates.k8s.io/v1beta1
depracated.
i got ant issue with creating CSR.
by using your script
https://github.com/doitintl/kube-secrets-init/blob/master/deployment/webhook-create-signed-cert.sh
i can approve it, but it is not issued.
maybe you can help my on this. please
Hi, would you be able to create a release of version 0.4.3/upload the docker image?
admissionregistration.k8s.io/v1beta1 is not available anymore and now we need to use admissionregistration.k8s.io/v1
I tried to change it manually but got errors like :
"Internal error occurred: failed calling webhook "secrets-init.doit-intl.com": Post https://secrets-init-webhook-svc.default.svc:443/pods?timeout=30s: context deadline exceeded"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.