kyma-project / warden Goto Github PK
View Code? Open in Web Editor NEWImage authenticity validation made k8s-native
License: Apache License 2.0
Image authenticity validation made k8s-native
License: Apache License 2.0
I0203 11:51:12.809812 1 request.go:682] Waited for 1.047510565s due to client-side throttling, not priority and fairness, request: GET:https://api.c-9fc93f2.kyma-dev.internal.canary.k8s.ondemand.com:443/apis/crd.projectcalico.org/v1?timeout=32s
2023-02-03T11:51:13.395Z INFO certs/cert.go:50 ensuring webhook secret
2023-02-03T11:51:13.483Z INFO certs/cert.go:61 updating pre-exiting webhook secret
2023-02-03T11:51:17.098Z INFO resource-ctrl certs/controller.go:45 initializing the defaulting webhook configuration
2023-02-03T11:51:17.275Z INFO resource-ctrl certs/controller.go:50 initializing the validation webhook configuration
2023-02-03T11:51:17.451Z INFO resource-ctrl certs/controller.go:55 creating webhook resources controller
2023-02-03T11:51:17.451Z INFO admission/main.go:95 setting up webhook server
2023-02-03T11:51:17.451Z INFO admission/main.go:109 starting the controller-manager
2023-02-03T11:51:17.452Z INFO manager/internal.go:362 Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:9090"}
2023-02-03T11:51:17.452Z INFO controller/controller.go:185 Starting EventSource {"controller": "webhook-resources-controller", "source": "kind source: *v1.ValidatingWebhookConfiguration"}
2023-02-03T11:51:17.452Z INFO controller/controller.go:185 Starting EventSource {"controller": "webhook-resources-controller", "source": "kind source: *v1.MutatingWebhookConfiguration"}
2023-02-03T11:51:17.452Z INFO controller/controller.go:185 Starting EventSource {"controller": "webhook-resources-controller", "source": "kind source: *v1.Secret"}
2023-02-03T11:51:17.452Z INFO controller/controller.go:193 Starting Controller {"controller": "webhook-resources-controller"}
2023-02-03T11:51:17.653Z INFO controller/controller.go:227 Starting workers {"controller": "webhook-resources-controller", "worker count": 1}
2023-02-03T11:51:17.654Z INFO webhook-resource-controller certs/controller.go:106 reconciling webhook resources {"name": "validation.webhook.warden.kyma-project.io"}
2023-02-03T11:51:17.654Z INFO webhook-resource-controller certs/controller.go:125 reconciling webhook validating webhook configuration
2023-02-03T11:51:17.742Z INFO certs/controller.go:134 reconciling webhook secret {"controller": "webhook-resources-controller", "object": {"name":"validation.webhook.warden.kyma-project.io"}, "namespace": "", "name": "validation.webhook.warden.kyma-project.io", "reconcileID": "3ffc1e09-fdce-492e-9890-56defb33f5da"}
2023-02-03T11:51:17.742Z INFO webhook-resource-controller certs/controller.go:113 webhook resources reconciled successfully {"name": "validation.webhook.warden.kyma-project.io"}
2023-02-03T11:51:17.743Z INFO webhook-resource-controller certs/controller.go:106 reconciling webhook resources {"name": "defaulting.webhook.warden.kyma-project.io"}
2023-02-03T11:51:17.743Z INFO webhook-resource-controller certs/controller.go:119 reconciling webhook defaulting webhook configuration
2023-02-03T11:51:17.832Z INFO certs/controller.go:134 reconciling webhook secret {"controller": "webhook-resources-controller", "object": {"name":"defaulting.webhook.warden.kyma-project.io"}, "namespace": "", "name": "defaulting.webhook.warden.kyma-project.io", "reconcileID": "aa4ace01-2838-4388-84ee-a5f3e259a3e5"}
2023-02-03T11:51:17.832Z INFO webhook-resource-controller certs/controller.go:113 webhook resources reconciled successfully {"name": "defaulting.webhook.warden.kyma-project.io"}
2023-02-03T11:51:17.832Z INFO webhook-resource-controller certs/controller.go:106 reconciling webhook resources {"name": "warden-admission-cert"}
2023-02-03T11:51:17.832Z INFO certs/controller.go:134 reconciling webhook secret {"controller": "webhook-resources-controller", "object": {"name":"warden-admission-cert","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "warden-admission-cert", "reconcileID": "373f00ed-6b09-4190-9b4b-eca1301c10fa"}
2023-02-03T11:51:17.833Z INFO webhook-resource-controller certs/cert.go:50 ensuring webhook secret
2023-02-03T11:51:17.833Z INFO webhook-resource-controller certs/cert.go:61 updating pre-exiting webhook secret
2023-02-03T11:51:17.833Z INFO webhook-resource-controller certs/controller.go:113 webhook resources reconciled successfully {"name": "warden-admission-cert"}
2023-02-03T11:52:04.924Z INFO admission/defaulting.go:91 pod was validated: nginx, kyma-system {"webhook": "defaulting"}
2023-02-03T11:52:15.024Z INFO admission/defaulting.go:91 pod was validated: warden-admission-7f7455774-jlbrf, kyma-system {"webhook": "defaulting"}
2023-02-03T11:52:20.958Z INFO admission/defaulting.go:91 pod was validated: nginx, kyma-system {"webhook": "defaulting"}
2023-02-03T11:52:35.028Z INFO admission/defaulting.go:91 pod was validated: serverless-webhook-svc-7c67445688-kbp44, kyma-system {"webhook": "defaulting"}
2023-02-03T11:52:52.754Z INFO admission/defaulting.go:91 pod was validated: serverless-docker-registry-self-signed-cert-g92kc, kyma-system {"webhook": "defaulting"}
2023-02-03T11:53:15.888Z INFO admission/defaulting.go:91 pod was validated: monitoring-prometheus-node-exporter-vm42x, kyma-system {"webhook": "defaulting"}
2023-02-03T11:53:57.445Z INFO admission/defaulting.go:91 pod was validated: monitoring-prometheus-node-exporter-7q85l, kyma-system {"webhook": "defaulting"}
2023-02-03T11:54:11.537Z INFO admission/defaulting.go:91 pod was validated: api-gateway-init-djsfc, kyma-system {"webhook": "defaulting"}
2023-02-03T11:54:33.763Z INFO admission/defaulting.go:91 pod was validated: nginx, kyma-system {"webhook": "defaulting"}
2023-02-03T11:54:39.466Z INFO admission/defaulting.go:91 pod was validated: nginx, kyma-system {"webhook": "defaulting"}
2023-02-03T11:54:45.594Z INFO admission/defaulting.go:91 pod was validated: nginx, kyma-system {"webhook": "defaulting"}
2023-02-03T11:54:50.337Z INFO admission/defaulting.go:91 pod was validated: nginx, kyma-system {"webhook": "defaulting"}
2023-02-03T11:54:51.370Z INFO admission/defaulting.go:91 pod was validated: warden-operator-686548bc49-jqqx2, kyma-system {"webhook": "defaulting"}
2023-02-03T11:54:55.634Z INFO admission/defaulting.go:91 pod was validated: nginx, kyma-system {"webhook": "defaulting"}
2023-02-03T11:54:59.781Z INFO admission/defaulting.go:91 pod was validated: monitoring-prometheus-node-exporter-9m67h, kyma-system {"webhook": "defaulting"}
2023-02-03T11:55:18.384Z INFO admission/defaulting.go:91 pod was validated: serverless-ctrl-mngr-76688cc688-w5l64, kyma-system {"webhook": "defaulting"}
2023-02-03T11:55:23.804Z INFO admission/defaulting.go:91 pod was validated: serverless-docker-registry-self-signed-cert-8xkf4, kyma-system {"webhook": "defaulting"}
2023-02-03T11:55:25.966Z INFO admission/defaulting.go:91 pod was validated: warden-admission-7f7455774-jlbrf, kyma-system {"webhook": "defaulting"}
2023-02-03T11:55:31.559Z INFO admission/defaulting.go:91 pod was validated: serverless-webhook-svc-7c67445688-kbp44, kyma-system {"webhook": "defaulting"}
2023-02-03T11:55:37.110Z INFO admission/defaulting.go:91 pod was validated: serverless-docker-registry-self-signed-cert-g92kc, kyma-system {"webhook": "defaulting"}
2023-02-03T11:55:39.302Z INFO admission/defaulting.go:91 pod was validated: monitoring-prometheus-node-exporter-vm42x, kyma-system {"webhook": "defaulting"}
2023-02-03T11:55:42.534Z INFO admission/defaulting.go:91 pod was validated: monitoring-prometheus-node-exporter-7q85l, kyma-system {"webhook": "defaulting"}
2023-02-03T11:55:45.801Z INFO admission/defaulting.go:91 pod was validated: api-gateway-init-djsfc, kyma-system {"webhook": "defaulting"}
2023-02-03T11:55:48.930Z INFO admission/defaulting.go:91 pod was validated: warden-operator-686548bc49-jqqx2, kyma-system {"webhook": "defaulting"}
2023-02-03T11:55:52.310Z INFO admission/defaulting.go:91 pod was validated: monitoring-prometheus-node-exporter-9m67h, kyma-system {"webhook": "defaulting"}
2023-02-03T11:55:55.418Z INFO admission/defaulting.go:91 pod was validated: serverless-ctrl-mngr-76688cc688-w5l64, kyma-system {"webhook": "defaulting"}
2023-02-03T11:56:01.436Z INFO admission/defaulting.go:91 pod was validated: serverless-docker-registry-self-signed-cert-8xkf4, kyma-system {"webhook": "defaulting"}
I0203 12:09:19.025674 1 request.go:682] Waited for 1.04373771s due to client-side throttling, not priority and fairness, request: GET:https://api.c-9fc93f2.kyma-dev.internal.canary.k8s.ondemand.com:443/apis/serverless.kyma-project.io/v1alpha1?timeout=32s
1.6754261596133406e+09 INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"}
1.6754261596138375e+09 INFO setup starting manager
1.6754261596146038e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.6754261596150367e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
I0203 12:09:19.614910 1 leaderelection.go:248] attempting to acquire leader lease kyma-system/c3790980.warden.kyma-project.io...
I0203 12:09:37.301316 1 leaderelection.go:258] successfully acquired lease kyma-system/c3790980.warden.kyma-project.io
1.6754261773014004e+09 DEBUG events warden-operator-686548bc49-n2jpf_c6b07a48-1e01-44ba-832b-eb4bee9634f1 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"kyma-system","name":"c3790980.warden.kyma-project.io","uid":"7a799a3d-42e3-4229-9336-973bcd38d070","apiVersion":"coordination.k8s.io/v1","resourceVersion":"113060"}, "reason": "LeaderElection"}
1.675426177301783e+09 INFO Starting EventSource {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
1.6754261773018472e+09 INFO Starting Controller {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod"}
1.6754261779046865e+09 INFO Starting workers {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
1.6754261807763028e+09 INFO No valid trust data for 11.18-alpine3.17 {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"ory-postgresql-0","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "ory-postgresql-0", "reconcileID": "a4f109be-d364-48a8-bbb7-f83a627a8a56"}
1.6754261837436259e+09 INFO pod validation failed {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"ory-postgresql-0","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "ory-postgresql-0", "reconcileID": "a4f109be-d364-48a8-bbb7-f83a627a8a56", "name": "ory-postgresql-0", "namespace": "kyma-system"}
1.6754261862038777e+09 INFO No valid trust data for 1.16.1-distroless {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"logging-loki-0","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "logging-loki-0", "reconcileID": "29dbe555-925c-4e8c-965a-6d8a061f8746"}
1.6754261887441628e+09 INFO No valid trust data for 2.2.1-9d10ad20 {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"logging-loki-0","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "logging-loki-0", "reconcileID": "29dbe555-925c-4e8c-965a-6d8a061f8746"}
1.6754261913826005e+09 INFO pod validation failed {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"logging-loki-0","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "logging-loki-0", "reconcileID": "29dbe555-925c-4e8c-965a-6d8a061f8746", "name": "logging-loki-0", "namespace": "kyma-system"}
1.6754261995073507e+09 INFO pod validated successfully {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"eventing-nats-2","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "eventing-nats-2", "reconcileID": "439ce8bf-c998-4b23-9423-93a95df2286a", "name": "eventing-nats-2", "namespace": "kyma-system"}
1.6754262038848813e+09 INFO pod validated successfully {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"eventing-nats-0","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "eventing-nats-0", "reconcileID": "b0123762-1ab9-49d8-962c-11144da5965a", "name": "eventing-nats-0", "namespace": "kyma-system"}
1.6754262096175299e+09 INFO No valid trust data for v2.40.7 {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"monitoring-prometheus-istio-server-6dd6c95cdd-hl9cc","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "monitoring-prometheus-istio-server-6dd6c95cdd-hl9cc", "reconcileID": "ff0d8288-ea8a-4e15-8281-638ea55cbdc7"}
1.6754262096175737e+09 INFO pod validation failed {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"monitoring-prometheus-istio-server-6dd6c95cdd-hl9cc","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "monitoring-prometheus-istio-server-6dd6c95cdd-hl9cc", "reconcileID": "ff0d8288-ea8a-4e15-8281-638ea55cbdc7", "name": "monitoring-prometheus-istio-server-6dd6c95cdd-hl9cc", "namespace": "kyma-system"}
1.675426211957088e+09 INFO No valid trust data for v20230111-13238c0f {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"telemetry-operator-66f45f6987-mbczh","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "telemetry-operator-66f45f6987-mbczh", "reconcileID": "d6e8f7ce-67c4-41a4-a500-66e259eb347a"}
1.675426213151528e+09 INFO signing.repositories.cloud.sap does not have trust data for eu.gcr.io/kyma-project/webhook-cert-init {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"telemetry-operator-66f45f6987-mbczh","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "telemetry-operator-66f45f6987-mbczh", "reconcileID": "d6e8f7ce-67c4-41a4-a500-66e259eb347a"}
1.675426213151582e+09 INFO pod validation failed {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"telemetry-operator-66f45f6987-mbczh","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "telemetry-operator-66f45f6987-mbczh", "reconcileID": "d6e8f7ce-67c4-41a4-a500-66e259eb347a", "name": "telemetry-operator-66f45f6987-mbczh", "namespace": "kyma-system"}
1.6754262143657887e+09 INFO No valid trust data for 1.16.1-distroless {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"nginx","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "nginx", "reconcileID": "0d680608-479d-4e14-a0f0-d205dbdc9d7a"}
1.6754262157542026e+09 INFO signing.repositories.cloud.sap does not have trust data for nginx {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"nginx","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "nginx", "reconcileID": "0d680608-479d-4e14-a0f0-d205dbdc9d7a"}
1.6754262157542615e+09 INFO pod validation failed {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"nginx","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "nginx", "reconcileID": "0d680608-479d-4e14-a0f0-d205dbdc9d7a", "name": "nginx", "namespace": "kyma-system"}
1.6754262169204524e+09 INFO signing.repositories.cloud.sap does not have trust data for victoriametrics/vmagent {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"rma-victoria-metrics-agent-5986d4b86d-pzw9l","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "rma-victoria-metrics-agent-5986d4b86d-pzw9l", "reconcileID": "f32ca999-9829-4495-8d68-555aba6598f6"}
1.6754262169204886e+09 INFO pod validation failed {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"rma-victoria-metrics-agent-5986d4b86d-pzw9l","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "rma-victoria-metrics-agent-5986d4b86d-pzw9l", "reconcileID": "f32ca999-9829-4495-8d68-555aba6598f6", "name": "rma-victoria-metrics-agent-5986d4b86d-pzw9l", "namespace": "kyma-system"}
1.67542621813184e+09 INFO No valid trust data for 1.16.1-distroless {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"central-application-gateway-7f866b9d9-nxt85","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "central-application-gateway-7f866b9d9-nxt85", "reconcileID": "0250f00d-b677-4c09-bb4c-be8cb8d1849a"}
1.6754262206627007e+09 INFO No valid trust data for v20221227-2dddc00f {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"central-application-gateway-7f866b9d9-nxt85","namespace":"kyma-system"}, "namespace": "kyma-system", "name": "central-application-gateway-7f866b9d9-nxt85", "reconcileID": "0250f00d-b677-4c09-bb4c-be8cb8d1849a"}
1.6754262206627467e+09 INFO pod validation failed {"controller": "pod", "controllerGro...
Once the architectural challenges for v0 are fulfilled, we need to create a helm chart and host it either as .tgz file or in the helm repo.
/kind feature
Warden defaulting webhook always reject pod request when notary service is not available.
Fix:
Description
we need additional control-loop that will engage/disable signature verification of image signatures for pods in a given namespace whenever a control-label ("protected namespace marker") is introduced/removed in the namespace
Reason
It is needed to mitigate the situation when user removes label from protected namespace ( i.e kyma-system) and attempts to deploy an image afterwards. In theory he can deploy a malicious image in the time period until the namespace label is back.
Description
Warden has a filesystem cache for calls towards notary service.
We should make sure that the cache is shared among the two modes of operation
Reason
New admission controller with be hooked into pod lifecycle in the kubernetes and will increase the times the image signature is checked. We should not increase the number of external calls to signify by introducing admission controller but reuse the cached results.
Otherwise it cannot setup communication with kubernetes and errors with TLS hanshade error
Description
Expected response time varies between different landscapes
(Signify service intend to introduce rate limit on Dev) and a 2 seconds timeout may be fine for PROD but will not be sufficient for DEV and STAGE
AC
We have currently authorization to Notary in pkg/validate/notary.go
Now what's missing is lack of logic that allows to validate the specific image reference with its tag against this notary service.
no trust data found for %image%
%image% doesn't have verified signature
/assign @koala7659
/kind feature
The pod should be reconciled if the following requirements are met:
Every time any of the validations is valid, the reconcile function needs to connec to the notary, validate the image signatures and if any of them fails -> set pod validation label to failed and requeue reconciliation after 5-10 minutes.
If the validation result is different than current state, update the label in the pod.
/assign @muralov
/kind feature
Description
To make sure warden is calling a trusted notary service, it needs to validate it's authenticity using a configured public key.
Reason
Protect image verification against fake notary services
Improve logging in warden project:
allowed-registry
admission webhook is rejecting pod update initiated by signify operator (failed verification case)
When pod is labeled "pending" and the namespace controller sets "pending" again, the pod controller sees no difference and doesn't validate the pod. We should think about how namespace controllers should set labels when namespace validation is enabled.
/kind bug
Upon further inspection of Signify access logs I've managed to find a problem where Warden's reconciliation loop would cause unintended DoS attack when it asks for unsigned images.
Once every 10 minutes, for a failed
image the retry reconciliation is issued. This currently occurs for every instance where Warden is deployed.
warden/controllers/pod_controller.go
Line 81 in b628532
Cache is not working for unsigned images, because Warden only caches signatures for already signed images.
Possible solutions:
Description
Configure CI job to run the integration tests via dedicated make target make run-integration-test
AC
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.