Comments (26)
My temporary solution:
kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')
from sealed-secrets.
I found this and tried the "hack" i keep crashing tho for an AWS EKS cluster.
> kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')
panic: Error fetching certificate: Unauthorized
goroutine 1 [running]:
main.main()
/home/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9
from sealed-secrets.
going to cut a new release soon
from sealed-secrets.
Due to the fact that I have Weave CNI installed in my EKS cluster I seem to have broken the ability for kubectl proxy
to be able to communicate to pods in my cluster. So if I run kubectl proxy
and then browse to http://localhost:8001/api/v1/namespaces/kube-system/services/sealed-secrets-controller/proxy/healthz I get the following response:
Error: 'Address is not allowed'
Trying to reach: 'http://10.32.0.3:8080/healthz'
It seems like kubeseal --fetch-cert
is relying on this functionality to work. The error from kubeseal I get is:
$ kubeseal --fetch-cert
panic: Error fetching certificate: the server is currently unable to handle the request (get services http:sealed-secrets-controller:)
goroutine 1 [running]:
main.main()
/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:340 +0x5c6
Currently the work around that I have gotten to work is:
$ kubectl port-forward -n kube-system svc/sealed-secrets-controller 8080
$ curl -LJO http://localhost:8080/v1/cert.pem
This can also be done in a single shell using something like the following. This was useful to me as I am trying to retrieve the public key within a CI pipeline.
# inspired by https://stackoverflow.com/a/54931055
kubectl port-forward -n kube-system svc/sealed-secrets-controller 8080 &
proxy_pid=$!
echo $proxy_pid
until curl -fsSL http://localhost:8080/healthz > /dev/null; do
echo "waiting for kubectl port-forward" >&2
sleep 2
done
curl -LJO http://localhost:8080/v1/cert.pem
function cleanup {
echo "killing kubectl proxy" >&2
kill $proxy_pid
}
trap cleanup EXIT
Just posting this here in case it is useful to anyone else.
from sealed-secrets.
Interesting, thanks for the report.
As background explanation: --fetch-cert
requires reading the public-key portion of the sealed-secrets master Secret in kube-system
somehow. Accessing the Secret
directly works, but is a bit scary security-wise (since the private-key is there too), so instead I serve just the public-key portion via HTTP from the sealed-secrets controller, and then kubeseal
fetches it via a proxy through the k8s apiserver. This is why it requires services/proxy
in kube-system
.
So, your options are:
- Retry with a k8s user with greater access. "
system:anonymous
" sounds like you are using an unauthenticated user of some sort. - If you can read the logs from the sealed-secrets-controller pod (
kubectl -n kube-system logs sealed-secrets-controller-5bc5488bb4-79blg
in your example above), then the public key is written to the log on startup(*). I expectsystem:anonymous
will not grant sufficient access for this either. - Get to the
http://sealed-secrets-controller.kube-system:8080/v1/cert.pem
in-cluster URL some other way, perhaps from a pod running in the cluster.
(*) The public key (via any method) is the text between and including -----BEGIN CERTIFICATE-----
and -----END CERTIFICATE-----
It doesn't matter which user you use to get the public key, it's the same public key for all users of that cluster. So typically the workflow would be: admin installs sealed-secret controller, admin runs kubeseal --fetch-cert
, admin publishes cert via some other means (intranet webpage, git repo, etc), regular users use cert.
For this bug:
It would be nice if we could make this easier for future users of EKS. I haven't used EKS - how did you get these credentials to the EKS cluster? Is it going to be typical for EKS users to use system:anonymous
in this way? If so, we might need to explicitly add a more liberal services/proxy
RBAC rule for the controller http service (it is meant to be ~widely accessible).
from sealed-secrets.
just for information:
This issue obviously also pops up when you're trying to seal a secret:
kubeseal < secrets.yaml
Please enter Username: Please enter Password: **panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"
goroutine 1 [running]:
main.main()
/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9
I wasted a lot of time to find out what the problem is here, until I found this issue. Maybe you could mention the workaround in the README?
from sealed-secrets.
Hi guys, do you have any ETA for updating client-go lib and rebuild?
Or maybe you have some beta binaries?
And one more thing - can I set controller IP manually? It tries to connect directly to pod.
admssa$ kubeseal --fetch-cert --token $(aws-iam-authenticator token -i cluster-name | jq -r '.status.token')
panic: Error fetching certificate: an error on the server ("Error: 'dial tcp 10.237.72.52:8080: getsockopt: connection timed out'\nTrying to reach: 'http://10.237.72.52:8080/v1/cert.pem'") has prevented the request from succeeding (get services http:sealed-secrets-controller:)
goroutine 1 [running]:
from sealed-secrets.
Same issue here:
panic: Error fetching certificate: Unauthorized
goroutine 1 [running]:
main.main()
/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9
from sealed-secrets.
In case anyone else comes across this issue, besides all the tremendously useful answers above, here is a practical summary of what to look for in order to get this going:
So #126 seems to have worked, however, this is NOT included in the v0.7.0 release of kubeseal (see commit history). So to have kubeseal --fetch-cert
work, you may need to clone the repo and make kubeseal
.
The second catch, is that kubeseal --fetch-cert
uses a Kubernetes service to communicate with the sealed-secrets controller and get the cert from /v1/cert.pem
(see comment right above this one). If for some reason you get an Error fetching certificate: services "sealed-secret-controller" not found
then your service may be named something else (i.e., sealed-secrets
).
So in short, you may have to clone the repo, make kubeseal
, use that executable instead of the one in the release and make sure you are using the right service name.
In my case, I used the official Helm chart to deploy sealed-secrets and ended up with this:
$ kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 15h
sealed-secrets ClusterIP 172.20.241.81 <none> 8080/TCP 14h
tiller-deploy ClusterIP 172.20.235.94 <none> 44134/TCP 14h
So I had to do
$ ./kubeseal --fetch-cert --controller-name sealed-secrets
-----BEGIN CERTIFICATE-----
...
to get the cert.
from sealed-secrets.
Would be pretty cool if there could be a new release cut with #126 included to resolve the issues of fetching the cert w/kubectl proxy. This also affects encrypting secrets (must download cert locally- and --fetch-cert
is broken too). Last release was over 1 year ago. Cloning the repo and running make kubeseal
does not work due to missing go dependencies.
For reference, this is the error I received:
panic: Error fetching certificate: an error on the server ("Error: 'dial tcp 10.65.1.2:8080: i/o timeout'\nTrying to reach: 'http://10.65.1.2:8080/v1/cert.pem'") has prevented the request from succeeding (get services http:sealed-secrets-controller:)
I'm now writing a workaround into my wrapper to use port-forwarding to fetch the cert, save it locally, use it for encryption, then remove the cert, instead of just being able to run kubeseal <mysecret.json >mysealedsecret.json
This did work as expected when I was using a cluster with Public IP's. Now that I am using a cluster with Private IP's, it is not, and I have to do a couple extra steps.
from sealed-secrets.
My kubeconfig is setup using eksctl
:
eksctl utils write-kubeconfig my-cluster --profile my-profile
Relevant code is here: https://github.com/weaveworks/eksctl/blob/master/cmd/eksctl/utils.go#L85-L164
EKS requires aws-iam-authenticator
to auth with k8s. It was formerly called heptio-authenticator-aws
and is vendored in eksutil
here: https://github.com/weaveworks/eksctl/tree/master/vendor/github.com/heptio/authenticator/cmd/heptio-authenticator-aws.
from sealed-secrets.
I'm able to view logs for the controller pod and get the certificate that way. I'll try updating RBAC tomorrow.
from sealed-secrets.
Huh, I think I originally misunderstood the issue here. This isn't about RBAC, since we should not allow system:anonymous
to access the service (there's nothing actually wrong with this since the pubkey is supposed to be public, but system:anonymous
represents API requests that have not provided any authentication token and probably should not have any access to the cluster at all). As @jramos last comment shows, if we provide a valid token, then we're no longer system:anonymous
and everything works.
So the issue here is that the API request that kubeseal
makes does not include the AWS bearer token. The fix appears to require updating to a version of client-go that includes the "exec" auth plugin, which is alpha in 1.10 / beta in k8s 1.11.
Meanwhile, the correct workaround is the explicit --token
arg from the previous comment.
from sealed-secrets.
👍 thanks for the explanation.
from sealed-secrets.
can I set controller IP manually?
No. As currently written, kubeseal
talks to the controller via the apiserver "proxy" to a known service (by namespace+name).
You only need this for fetching the public key though - you can run kubeseal --fetch-cert
once, and then pass the certificate file to later invocations of kubeseal --cert
. If you do this, then kubeseal
doesn't talk to the controller at all during sealing. This is the recommended approach for automated pipelines (since it removes the need to give your robots authenticated k8s API access).
from sealed-secrets.
It looks like this has been resolved with #126. I just tested on an EKS cluster and it is working correctly for me.
from sealed-secrets.
I'm getting same error as @NeoTech
EDIT: Managed workaround. Use local secret: kubeseal --cert sealed-secrets-key.yaml --fetch-cert
EDIT2: Still not working. output is supposed to be a pem file but I'm getting yaml output. But my workaround to use entirely local workflow is valid just --fetch-cert doesn't work correctly
from sealed-secrets.
kubectl port-forward sealed-secrets-controller-xxxxx 8080:8080
and then
curl -O localhost:8080/v1/cert.pem
is doing the trick for me! from then on use the local file to seal secrets
from sealed-secrets.
As an extra comment, I got so involved in debugging this that I forgot what the original issue is:
Originally, if I wanted to do seal a secret, I got the following
$ kubeseal <mysecret.json >mysealedsecret.json
panic: Error fetching certificate: services "http:sealed-secrets:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"
goroutine 1 [running]:
main.main()
/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9
Again, by cloning and compiling the kubeseal binary myself (again, this inlcudes the bump to v0.7.0 of client-go) I was the able to seal a secret as normal:
$ ./kubeseal --controller-name sealed-secrets <mysecret.json >mysealedsecret.json
from sealed-secrets.
@alejandrox1 When do we expect an official release?
from sealed-secrets.
@mbelang I don't know 😅 I'm not actually a maintainer of this project.
from sealed-secrets.
@jgrabenstein do you use kubeseal the same way you use kubectl (or any other Kubernetes client)?
Could you provide more details about how your cluster is setup? Source code if possible to try and replicate it.
from sealed-secrets.
Getting this when running on EKS Kubernetes version 1.12.
kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')
panic: Error fetching certificate: Unauthorized
goroutine 1 [running]:
main.main()
/private/tmp/kubeseal-20181221-52005-1n89slr/sealed-secrets-0.7.0/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3ad
Does anyone have an update?
from sealed-secrets.
If you use the git repo from master it works. They haven't done a release in a long time. Is this project still being maintained?
from sealed-secrets.
@jgrabenstein do you use kubeseal the same way you use kubectl (or any other Kubernetes client)?
Could you provide more details about how your cluster is setup? Source code if possible to try and replicate it.
These are regional, private (no external/public IP's) GKE clusters on a shared VPC.
It appears that running simple commands like kubeseal <mysecret.json >mysealedsecret.json
do not work with private clusters (see the error I reported above).
To work around that, I use kubectl port-forward
to make the controller's port available locally. Then i can download the cert from the controller. Next you will encrypt with kubeseal, pointing it to the cert you just downloaded: kubeseal --cert /path/to/public-cert $RAW_SECRET > $SEALED_SECRET
. You can then apply the sealed secret as you normally would.
Not sure my company would appreciate me sharing the source code for my wrapper.. If people here think it would help them I could possibly see if we could contribute it.
I think ultimately this project needs some decision/leadership around its future - this is something that could be fixed in kubeseal
, even if it cannibalizes my wrapper's method.
from sealed-secrets.
Would it make sense if the sealedsecret controller posted the public key into a configmap?
from sealed-secrets.
Related Issues (20)
- Template `data` field treated as `stringData`, `stringData` ignored HOT 4
- Kubeseal not working on EKS IPv6 CLuster HOT 8
- kubeseal does not respond HOT 7
- Document system:authenticated group usage in GKE HOT 3
- Kubeseal is not able to fetch certificate HOT 1
- Kubeseal --re-encrypt with local certificates HOT 1
- Activate the golangci-lint gosec linter in the project configuration HOT 1
- Program managed sealed secret in GitOps environment (using ArgoCD)
- [Helm] Netpol & Servicemonitor enabled - not able to scrape metrics HOT 1
- lastTransitionTime/lastUpdateTime under status->conditions not updated when updating a SealedSecret
- kubeseal is ignoring NO_PROXY variables upon request for cert.pem (403) HOT 4
- Unseal with --raw HOT 6
- Secret values not getting updated HOT 1
- Vendoring fails on 0.26.1 HOT 1
- Bug in Helm Chart in file role-binding.yaml
- Ability to enable TLS on Ingress with custom certificate which exists on the cluster already HOT 4
- Creating sealed secret throws an error: "an empty namespace may not be set when a resource name is provided" HOT 1
- Status shows no key could decrypt secret for successful created secret HOT 1
- decrypt secrets encrypted with custom TLS certificate of expired validity HOT 1
- Secret not yet active when deploying with Helm HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sealed-secrets.