kapicorp / tesoro Goto Github PK
View Code? Open in Web Editor NEWKapitan Admission Controller Webhook for secrets mgmt
License: Apache License 2.0
Kapitan Admission Controller Webhook for secrets mgmt
License: Apache License 2.0
TL;DR
Although the design of tesoro
would allow it to run as non-root user, we are forced to run as root for a number of reasons.
Reason
Ideally, we would want to run tesoro
on port 8443
(unprivileged) and then user the service to expose it as port 443
.
This works on both KIND and GKE "non private" clusters.
Unfortunately on private GKE clusters, although the apiserver
is meant to connect to the tesoro webhook using the service IP on port 443, the connection seems to then reach the pod IP directly, expecting tesoro to be running on the same port.
Logs:
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
tesoro ClusterIP 10.15.5.59 <none> 443/TCP,9095/TCP 52s
$ kubectl apply -f example-gkms-secret.yml
Error from server (InternalError): error when creating "example-gkms-secret.yml": Internal error occurred: failed calling webhook "tesoro.kapicorp.com
": Post https://tesoro.tesoro.svc:443/mutate?timeout=30s: no service port '\u20fb' found for service "tesoro"
We've also experienced other kind of issues like
$ kubectl apply -f example-gkms-secret.yml
Error from server (InternalError): error when creating "example-gkms-secret.yml": Internal error occurred: failed calling webhook "tesoro.kapicorp.com": Post https://tesoro.tesoro.svc:443/mutate?timeout=5s: dial tcp 10.16.0.18:8443: i/
o timeout
while this is probably because of firewall rules: https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1809
Error from server (InternalError): error when creating "example-gkms-secret.yml": Internal error occurred: failed calling webhook "tesoro.kapicorp.com": Post https://tesoro.tesoro.svc:443/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Because of this kubernetes/kubernetes#61510 although the api supports it, we cannot run both service and pod on 8443 either.
The only solution that seems to work is to have the actual tesoro
pod also running on port 443, which makes it work instantly :(
Describe the bug/feature
Mutating webhook throws the following error when submitting the valid nginx example:
$ kubectl apply -f tests/k8s/nginx_deployment.yml
Error from server (InternalError): error when creating "tests/k8s/nginx_deployment.yml": Internal error occurred: failed calling webhook "tesoro-admission-controller.tesoro.svc": Post "https://tesoro-admission-controller.tesoro.svc:443/mutate?timeout=30s": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
To Reproduce
I am evaluating Tesoro so was following the instructions:
$ minikube start
๐ minikube v1.15.1 on Ubuntu 18.04
โจ Using the docker driver based on user configuration
๐ Starting control plane node minikube in cluster minikube
๐ฅ Creating docker container (CPUs=2, Memory=3900MB) ...
๐ณ Preparing Kubernetes v1.19.4 on Docker 19.03.13 ...
๐ Verifying Kubernetes components...
๐ Enabled addons: storage-provisioner, default-storageclass
๐ Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
git clone [email protected]:kapicorp/tesoro.git
kubectl apply -f k8s/clusterrole.yaml
kubectl apply -f k8s/clusterrolebinding.yaml
kubectl apply -f k8s/tesoro_namespace.yaml
kubectl -n tesoro apply -f k8s/tesoro_secret.yaml
kubectl -n tesoro apply -f k8s/tesoro_service.yaml
kubectl -n tesoro apply -f k8s/tesoro_deployment.yaml
Wait for pods to start
kubectl apply -f k8s/tesoro_mutatingwebhook.yaml
Test failed
$ kubectl apply -f tests/k8s/nginx_deployment.yml
Error from server (InternalError): error when creating "tests/k8s/nginx_deployment.yml": Internal error occurred: failed calling webhook "tesoro-admission-controller.tesoro.svc": Post "https://tesoro-admission-controller.tesoro.svc:443/mutate?timeout=30s": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
Expected behavior
Expected example to work
Describe the bug/feature
Tesoro fails to detect and reveal secrets embedded into other strings
kind: Secret
metadata:
labels:
name: bug-report
tesoro.kapicorp.com: enabled
name: bug-report
stringData:
not_work: does not work ?{base64:eyJkYXRhIjogIk4wOTRlbE5XTlZWalFVZzRWblJKVTBscVpWWXpTVkZYV2pGWFVVOVViV1JSWlRFeFFYWlNhbVJuU1E9PSIsICJlbmNvZGluZyI6ICJvcmlnaW5hbCIsICJ0eXBlIjogImJhc2U2NCJ9:embedded}
works: ?{base64:eyJkYXRhIjogIk4wOTRlbE5XTlZWalFVZzRWblJKVTBscVpWWXpTVkZYV2pGWFVVOVViV1JSWlRFeFFYWlNhbVJuU1E9PSIsICJlbmNvZGluZyI6ICJvcmlnaW5hbCIsICJ0eXBlIjogImJhc2U2NCJ9:embedded}
results in:
{
"not_work": "ZG9lcyBub3Qgd29yayA/e2Jhc2U2NDpleUprWVhSaElqb2dJazR3T1RSbGJFNVhUbFpXYWxGVlp6UldibEpLVlRCc2NWcFdXWHBUVmtaWVYycEdXRlZWT1ZWaVYxSlNXbFJGZUZGWVdsTmhiVkp1VTFFOVBTSXNJQ0psYm1OdlpHbHVaeUk2SUNKdmNtbG5hVzVoYkNJc0lDSjBlWEJsSWpvZ0ltSmhjMlUyTkNKOTplbWJlZGRlZH0=",
"works": "N094elNWNVVjQUg4VnRJU0lqZVYzSVFXWjFXUU9UbWRRZTExQXZSamRnSQ=="
}
with annotations
k get secret/bug-report -o yaml | yq -r .metadata.annotations.'"tesoro.kapicorp.com/revealed"'
["/data/works"]
Hello,
I have observed the following behaviour already several times:
./compiled/dev/scripts/kubectl.sh apply -f compiled/dev/manifests/simcontrol-secret.yml
secret/simcontrol created
postgres_password
should be a plain password string, instead:./compiled/dev/scripts/kubectl.sh get secret simcontrol -o yaml | yq -r .data.postgres_password | base64 --decode
{
"keys": [
{
"use": "sig",
"kty": "OKP",
"crv": "Ed25519",
"alg": "EdDSA",
"x": "7NmW7vO2_ipJ8Miavm3rxBJaJqnmDjDnvFRBOaR-VhA",
"d": "L0DSyGASSVNwDdgutkpI8qGPlOk1o09jf8u4-o8J7Gw"
},
{
"use": "sig",
"kty": "EC",
"kid": "23ccf53f-fd72-461c-adb0-c5c4a60e37b7",
"crv": "P-256",
"alg": "ES256",
"x": "N2rXfvFT4S5GfJYyAFoyakFpMpLHzFUlwZMuy1x1O8E",
"y": "rdoijXCFISYI4lX9xwqu-dzK_Y6RurhZewgiQpdZkxk",
"d": "w-3cl7rJC_OZE_dzfCnrZw6UyUPKu21TMbSRpJDn5Es"
}
]
jwks_pub
:./compiled/dev/scripts/kubectl.sh get secret simcontrol -o yaml | yq -r .data.jwks_pub | base64 --decode
{
"keys": [
{
"use": "sig",
"kty": "OKP",
"crv": "Ed25519",
"alg": "EdDSA",
"x": "7NmW7vO2_ipJ8Miavm3rxBJaJqnmDjDnvFRBOaR-VhA"
},
{ "use": "sig",
"kty": "EC", "kid": "23ccf53f-fd72-461c-adb0-c5c4a60e37b7",
"crv": "P-256",
"alg": "ES256",
"x": "N2rXfvFT4S5GfJYyAFoyakFpMpLHzFUlwZMuy1x1O8E",
"y": "rdoijXCFISYI4lX9xwqu-dzK_Y6RurhZewgiQpdZkxk"
}
]
}
Also, revealing the secret manifest file directly with kapitan shows that the file has the correct content:
./kapitan.sh refs --reveal -f compiled/dev/manifests/simcontrol-secret.yml | yq -r .stringData.postgres_password | base64 --decode
REDACTED_CORRECT_PASSWORD
Restarting Tesoro (side effect of adding --verbose
to collect logs) and re-applying made it work correctly.
kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.9-gke.6", GitCommit:"14bc8ad5f8c245f1240a8e4eab128c4d51bfeffe", GitTreeState:"clean", BuildDate:"2020-05-30T02:07:03Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}
Currently --verbose
prints out decrypted secrets to stdout/logs.
This is dangerous because secrets would then leak into logs and would all need to be rotated.
I suggest the default version of --verbose
should attempt to redact secrets, and perhaps we should have an explicit redact_secrets=false
to enable printing out the secrets.
I believe encrypted should be allowed into logs, but this is also another one worth discussing given that, especially for non *KMS backends where the key rotates automatically, it could have security implications
Also,
2020-07-08 18:19:29 DEBUG Secret transformation: decoded_ref: ?{
would be more helpful with a key name
Describe the bug/feature
While looking at the logs, it is difficult to understand which actual kubernetes resource is being handled by kubernetes
To Reproduce
Steps to reproduce the behavior:
Expected behavior
a log line like
Namespace: alpha, name: my-secret, Kind: {'group': '', 'version': 'v1', 'kind': 'Secret'} Resource: {'group': '', 'version': 'v1', 'resource': 'secrets'}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.