Comments (15)
Ahh, interesting. I was having a dig around and I can see that the extension-apiserver-authentication ConfigMap in kube-system only has a client-ca-file key. Is there supposed to also be a requestheader-client-ca-file key too?
Our clusters (Tested MetaKube, AKS, EKS, GKE) do:
$ kubectl -n kube-system get cm extension-apiserver-authentication -ojson | jq -r '.data | keys | .[]'
client-ca-file
requestheader-allowed-names
requestheader-client-ca-file
requestheader-extra-headers-prefix
requestheader-group-headers
requestheader-username-headers
This might be the root cause. According to cert-manager/cert-manager#1149 which in turn references voyagermesh/voyager#888 (comment) you might have to enable the aggregation layer of the kube-apiserver component. The k8s documentation provides some insights into the flags needed, but maybe there is an easier way to enable that in your cluster depending on your way of setting the cluster up. Can you check whether the kube-apiserver is missing the flags mentioned in the docs and see if there is a way to provide those?
from cert-manager-webhook-dnsimple.
I'm not really sure what the error is indicating there. I couldn't really find anything while searching for it. Maybe the certificates for the front-proxy are not configured properly or the cert-manager pods that rely on the aggregated API server need to be restarted. If that does not fix the problem, maybe you can share the kube-apiserver flags you are using.
from cert-manager-webhook-dnsimple.
Hi @cmacrae, I waited a lot for this plug-in as well until it got to a point where we needed it so desperately in a project that I was willing to just give it a shot. The code needed was actually not complicated at all - sometimes you just need the proper motivation to write it 😉
The log line you attached is one I usually see in the plugin's log first until the client ca is seemingly injected after a few moments. Then everything works fine. Can you check that the cert-manager-webhook-dnsimple:webhook-authentication-reader
role binding exists in the kube-system
namespace, e.g.:
$ k get rolebinding cert-manager-webhook-dnsimple:webhook-authentication-reader -n kube-system
and if it does: can you elaborate on the environment (say GKE, AKS et al.) and the Kubernetes and cert-manager versions?
from cert-manager-webhook-dnsimple.
Thanks for the prompt reply @arnediekmann 👋
Yep, the role binding does exist :) Here's some info on my environment:
- On prem (at home... 🏠 ) deployment on the metal
- ArgoCD with a bunch of Helm charts for deployment
- cert-manager
0.15.2
chart deployed (all looks well) - cert-manager-webhook-dnsimple
0.0.3
chart deployed
cert-manager chart values
Just turning on CRD deployment (values are nested as I'm using requirements.yaml
)
cert-manager:
installCRDs: true
cert-manager-webhook-dnsimple chart values
(Same again with the nested values)
cert-manager-webhook-dnsimple:
image:
repository: neoskop/cert-manager-webhook-dnsimple
tag: 0.0.3
replicaCount: 1
dnsimple:
account: < REDACTED >
token: < REDACTED >
clusterIssuer:
staging:
enabled: true
email: < REDACTED >
Let me know if there's anything else I can provide to help debug :)
from cert-manager-webhook-dnsimple.
Not sure if these are any help, but here's an excerpt from cert-manager's ca-injector
logs:
I0825 09:10:07.470281 1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled" "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha1.acme.neoskop.de"}
I0825 09:10:13.498953 1 sources.go:94] cert-manager/inject-controller "msg"="Extracting CA from Certificate resource" "resource_kind"="APIService" "resource_name"="v1alpha1.acme.neoskop.de" "resource_namespace"="" "certificate"="cert-manager/cert-manager-webhook-dnsimple-webhook-tls"
I0825 09:10:13.503994 1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="APIService" "resource_name"="v1alpha1.acme.neoskop.de" "resource_namespace"="
So, it does look like it's doing its job
from cert-manager-webhook-dnsimple.
Hmm this does seem to look like the logs I receive. Take these for example:
W0804 15:49:25.854046 1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0804 15:49:25.854597 1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" due to: configmap "extension-apiserver-authentication" not found
I0804 15:49:25.944253 1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0804 15:49:25.944437 1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0804 15:49:25.944506 1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0804 15:49:25.944540 1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0804 15:49:25.946652 1 secure_serving.go:178] Serving securely on [::]:443
I0804 15:49:25.950501 1 dynamic_serving_content.go:129] Starting serving-cert::/tls/tls.crt::/tls/tls.key
I0804 15:49:25.951307 1 tlsconfig.go:219] Starting DynamicServingCertificateController
I0804 15:49:26.045053 1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0804 15:49:26.045122 1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
E0804 16:13:03.996198 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0804 16:13:03.996488 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0805 03:31:30.586676 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0805 03:31:30.587430 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0806 05:40:30.586246 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0806 05:40:30.586489 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0806 15:06:30.585648 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0806 15:06:30.585958 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0812 17:39:23.996129 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0812 17:39:23.996818 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0814 02:26:03.996227 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0814 02:26:03.997547 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0814 10:20:00.586300 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0814 10:20:00.586895 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0815 06:05:30.586838 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0815 06:05:30.587426 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0815 11:06:03.996372 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0815 11:06:03.997582 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0816 05:38:57.987483 1 webhook.go:197] Failed to make webhook authorizer request: context canceled
E0816 05:38:57.987834 1 errors.go:77] context canceled
E0817 15:55:00.586430 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0817 15:55:00.587011 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0820 16:43:31.613594 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0820 16:43:31.953637 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0820 17:25:34.330242 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0820 17:25:34.442271 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0820 18:24:54.013134 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0820 18:24:54.016331 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0822 07:25:15.709036 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0822 07:25:15.709907 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0822 07:25:15.710284 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0822 07:25:15.710469 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0822 17:59:30.585955 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0822 17:59:30.586596 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0824 08:52:58.568009 1 webhook.go:197] Failed to make webhook authorizer request: context canceled
E0824 08:52:58.568351 1 errors.go:77] context canceled
E0824 08:52:58.568775 1 webhook.go:197] Failed to make webhook authorizer request: context canceled
E0824 08:52:58.569046 1 errors.go:77] context canceled
E0824 08:52:58.569221 1 webhook.go:197] Failed to make webhook authorizer request: Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0824 08:52:58.569721 1 errors.go:77] Post https://10.240.16.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
However I always found the numerous errors a little concerning even though they don't seem to prevent the web hook from working properly. I think they emerge from the library provided by Jetstack, however. I will create a follow-up issue on their end and link the issue here.
from cert-manager-webhook-dnsimple.
Ahh I'm sorry, I just realized that you meant the CA injector was working properly, not the web hook. Nevertheless your setup looks very sound and shouldn't have any problems. I'm therefore not really sure where to start looking and will still create an issue in jetstack/cert-manager-webhook-example to enquire further and will get back to you!
from cert-manager-webhook-dnsimple.
No worries! Thanks very much 🙇 I'll be digging into the cert-manager-webhook-example code in the meantime to try and determine what the error means 😄
from cert-manager-webhook-dnsimple.
As far as I understand it, initially the plug-in will read the client-ca-file
field from the ConfigMap
kube-system/extension-apiserver-authentication
to talk to the Kubernetes control plane. The error seems to indicate that the service account of the pod (cert-manager-webhook-dnsimple
) is not allowed to read the ConfigMap
. This seems to happen randomly - as you can see in my logs - at first but should eventually start working shortly after startup. This got me thinking: maybe the role binding is not correct. Can you provide me with the following to dig deeper:
- The output of
kubectl get rolebinding cert-manager-webhook-dnsimple:webhook-authentication-reader -n kube-system -oyaml
- The major version of helm used
- The namespace where the cert-manager-webhook-dnsimple components where installed to
from cert-manager-webhook-dnsimple.
Ahh, interesting. I was having a dig around and I can see that the extension-apiserver-authentication
ConfigMap
in kube-system
only has a client-ca-file
key. Is there supposed to also be a requestheader-client-ca-file
key too? Just wondering as that's what the error message seems to be referring to (client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
).
Nevertheless, here's the info you requested:
Role binding output
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"RoleBinding","metadata":{"annotations":{},"labels":{"app":"cert-manager-webhook-dnsimple","app.kubernetes.io/instance":"cert-manager-webhook-dnsimple","chart":"cert-manager-webhook-dnsimple-0.0.4","heritage":"Tiller","release":"cert-manager-webhook-dnsimple"},"name":"cert-manager-webhook-dnsimple:webhook-authentication-reader","namespace":"kube-system"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"Role","name":"extension-apiserver-authentication-reader"},"subjects":[{"apiGroup":"","kind":"ServiceAccount","name":"cert-manager-webhook-dnsimple","namespace":"cert-manager"}]}
creationTimestamp: "2020-08-16T15:43:03Z"
labels:
app: cert-manager-webhook-dnsimple
app.kubernetes.io/instance: cert-manager-webhook-dnsimple
chart: cert-manager-webhook-dnsimple-0.0.4
heritage: Tiller
release: cert-manager-webhook-dnsimple
managedFields:
- apiVersion: rbac.authorization.k8s.io/v1beta1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:app: {}
f:app.kubernetes.io/instance: {}
f:chart: {}
f:heritage: {}
f:release: {}
f:roleRef:
f:apiGroup: {}
f:kind: {}
f:name: {}
f:subjects: {}
manager: argocd-application-controller
operation: Update
time: "2020-08-25T09:27:17Z"
name: cert-manager-webhook-dnsimple:webhook-authentication-reader
namespace: kube-system
resourceVersion: "4566455"
selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings/cert-manager-webhook-dnsimple:webhook-authentication-reader
uid: ede03b14-62b1-4021-af65-e1561c114241
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: cert-manager-webhook-dnsimple
namespace: cert-manager
Helm major version
Technically, I'm not "using" helm. ArgoCD has its own engine for interpreting Helm charts.
But, the answer is: 3
Namespace info
All cert-manager & cert-manager-webhook-dnsimple resources are deployed to the cert-manager
namespace
from cert-manager-webhook-dnsimple.
Thanks so much for the thorough research! The links really helped 🙌 I managed to get the API aggregation turned on, and the extension-apiserver-authentication
now has the requestheader-client-ca-file
in there :)
So, one step further! Here's where I am now:
I0825 16:22:23.274092 1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="argocd/argocd-ingress-3529246321-3537152352-882239069"
I0825 16:22:23.274342 1 dns.go:92] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="argo.< REDACTED >" "domain"="argo.< REDACTED >" "domain"="argo.cd-ingress-3529246321-3537152352-882239069" "resource_namespace"="argocd" "type"="dns-01"
E0825 16:22:23.278684 1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item due to error processing" "error"="dnsimple.acme.neoskop.de is forbidden: User \"front-proxy-client\" cannot create resource \"dnsimple\" in API group \"acme.neoskop.de\" at the cluster scope" "key"="argocd/argocd-ingress-3529246321-3537152352-882239069"
from cert-manager-webhook-dnsimple.
Got it working! I just needed to configure my cluster's auth settings
I can now see records being created in dnsimple! 🎉 Now I just have to figure out forwarding any _acme-challenge.*.my.domain
requests out to global DNS servers, rather than my internal servers responsible for that domain so they can resolve the challenge.
Thanks so much for this project and all your help @arnediekmann 🙏
from cert-manager-webhook-dnsimple.
Very cool! 🎊 I'm glad you got it working!
Now I just have to figure out forwarding any _acme-challenge.*.my.domain requests out to global DNS servers, rather than my internal servers responsible for that domain so they can resolve the challenge.
I found it easiest to just exclusively rely on DNSimple. But we are usually setting up clusters with Pulumi which allows me to grab the load balancer IPs from the cluster and create records in DNSimple programmatically.
from cert-manager-webhook-dnsimple.
Gotcha, that sounds like a nice setup 👍 All my stuff is on-prem (DNSimple is the first external service I'll be relying upon), so I'm doing self-hosted PowerDNS with external-dns handling record creation, which makes it the authoritave NS for my subdomain. But, I'm pretty sure I could get a setup with CoreDNS to match requests for _acme-challenge.*.my.domain
and only forward those out globally :)
We'll see! I'll be blogging about this stuff at some point, and of course I'll be mentioning this project 😁
from cert-manager-webhook-dnsimple.
Very interesting setup - haven't made much use of CoreDNS in Kubernetes / other DNS servers yet. Might be something worth considering. Will keep an eye out for that blog post 🙂
from cert-manager-webhook-dnsimple.
Related Issues (19)
- Staging cluster-issuer doesn't quotes the dnsimple account HOT 1
- DNS challenge failing on production HOT 5
- Add repository to helm hub HOT 2
- cert-manager-webhook-dnsimple throws errors on GKE running Kubernetes 1.17.15-gke.800 HOT 9
- Webhook pod having go panics HOT 1
- cert-manager deployment is not in the cert-manager namespace HOT 2
- Deprecation warnings HOT 2
- "Certificate" in version "cert-manager.io/v1alpha2" not available HOT 2
- no published docker image supporting arm (raspberry pi) HOT 4
- Not entirely sure what the groupName does HOT 1
- Container using privileged port 443 HOT 3
- Error: servfail - webhook service not doing anything
- Token creation HOT 1
- Working with willcard and dynamic branch
- Feature request: arm64 support HOT 3
- Helm Repository does not contain helm chart anymore HOT 1
- Dependency Dashboard
- OpenAPI spec for "v1alpha1.[groupName]" failed with: failed to download v1alpha1.[groupName]: resource not found HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cert-manager-webhook-dnsimple.