Kubescape helm install fails | kollector-0 pod crashes about kubescape HOT 12 CLOSED

Thank you @extensure123 for reporting.
Can you please try again at let me know if you encounter the same issue?

from kubescape.

Thanks @dwertent - Did we update anything in the helm charts since I raised this issue a few hours back?
I updated my kubescape helm repos and reinstalled the helm charts a moment back and now I see all the pods in a running state

root:~# kubectl get pods -n kubescape
NAME                         READY   STATUS    RESTARTS   AGE
kubescape-5d69c79464-xsv26   1/1     Running   0          26m
kubevuln-58cbbdd98f-w5k62    1/1     Running   0          26m
node-agent-8lng5             1/1     Running   0          26m
operator-7cd597c5dc-cfx2v    1/1     Running   0          26m
storage-6d87946d9d-pr48t     1/1     Running   0          26m

However, I do not see my cluster details reflected on the armosec.io dashboard so something is amiss.
So I reinstalled the helm charts a moment back with a --set server=api.armosec.io argument to the helm upgrade command to check if results reflect on my armosec.io dashboard account. It did not as now I see the discovery pods are not in a running state
root:~# kubectl get pods -n kubescape

NAME                         READY   STATUS       RESTARTS   AGE
kubescape-5d69c79464-xsv26   1/1     Running      0          34m
kubevuln-58cbbdd98f-w5k62    1/1     Running      0          34m
node-agent-8lng5             1/1     Running      0          34m
operator-7cd597c5dc-cfx2v    1/1     Running      0          34m
service-discovery-5z74n      0/1     Init:Error   0          4m35s
service-discovery-6q5n8      0/1     Init:Error   0          5m
service-discovery-cn8pb      0/1     Init:Error   0          3m50s
service-discovery-p2g9f      0/1     Init:Error   0          5m15s
service-discovery-vdkv6      0/1     Init:Error   0          2m24s
storage-6d87946d9d-pr48t     1/1     Running      0          34m

Am I missing something? Appreciate your inputs!!

from kubescape.

@amirmalka can you please take a look at this?

from kubescape.

@extensure123 Looking at the commands in the issue description I can see that you are trying to install the old helm chart (named kubescape-cloud-operator), which is deprecated.
The new helm chart is named kubescape-operator: https://github.com/kubescape/helm-charts/blob/main/charts/kubescape-operator/README.md

Unless there is a specific need, could you please try again with the new helm chart?
It would also be helpful, in case the issue is reproduced, to run the chart with --set logger.level=debug and attach the relevant logs.

from kubescape.

Thanks @amirmalka for pointing out on the deprecated old helm chart command that deploys kubescape-cloud-operator. My bad and sorry for the typo when I opened the issue.
Note: I had used kubescape-cloud-operator setup in the past (a few months ago) and things were working and integrated with the armosec.io dashboard UI. I think at that time I was using kubescape-cloud-operator version 1.10.8

I've reinstalled the new helm charts a moment back and using the kubescape-operator now based on the below commands (I'm using the same VM, same K8s version etc. based on earlier successful setup of the older version a few months back)

helm repo add kubescape https://kubescape.github.io/helm-charts/
helm repo update
helm upgrade --install kubescape kubescape/kubescape-operator -n kubescape --create-namespace --set clusterName=`kubectl config current-context` --set account=xxxx --set server=api.armosec.io --set logger.level=debug

The issue is still the same as my earlier update and comment.
"However, I do not see my cluster details reflected on the armosec.io dashboard so something is amiss. So I reinstalled the helm charts a moment back with a --set server=api.armosec.io argument to to the helm upgrade command (as suggested on the armosec dashboard so that my cluster gets connected to the UI) to re-check if my cluster is discovered and results reflect on my armosec.io dashboard account. It did not and now now I see the discovery pods are not in a running state"
Is there any specific outbound access to a specific URL required. The firewall at my end will block inbound to my K8s cluster.

from kubescape.

@extensure123 yes, the service discovery job of the kubescape operator tries to make an HTTP call (GET) to the following URL:
https://{{server}}/api/v1/servicediscovery

so in your case, when using api.armosec.io, make sure you don't block access to https://api.armosec.io/api/v1/servicediscovery as this endpoint will return the list of URLs needed in order to connect to this backend.

You can also understand the issue better by looking at the logs of the service discovery pods in case firewall is not the issue.

from kubescape.

The service-discovery-xxx pods are all in an Init:Error state and the logs of these pods only indicate the below

Defaulted container "update-configmap" out of: update-configmap, url-discovery (init)
Error from server (BadRequest): container "update-configmap" in pod "service-discovery-xxx" is waiting to start: PodInitializing

Seems curl https://api.armosec.io/api/v1/servicediscovery works fine from my on-prem VM running the kind cluster but once I get inside the kindest/node:v1.27.3 container running the K8s components, curl fails

root@kubescape-control-plane:/# curl https://api.armosec.io/api/v1/servicediscovery
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I'll check with the firewall team at my end. Hopefully there's no inbound request coming from kubescape SaaS to my cluster. Thanks again!!

from kubescape.

I manually installed my local CRT certificates inside my kind cluster as a workaround. Now curl to https://api.armosec.io/api/v1/servicediscovery endpoint works both from my VM and the from inside the single-node kind cluster. Now I installed helm charts again from scratch and it did get pass the the earlier "Init:Error" status of "service-discovery" pods throwing "Error: failed pre-install: timed out waiting for the condition".

However, this time, "gateway" pod fails to run

kubectl get pods -n kubescape
NAME                              READY   STATUS             RESTARTS      AGE
gateway-76c58b6569-lt6pd          0/1     CrashLoopBackOff   1 (19s ago)   111s
kollector-0                       1/1     Running            0             77s
kubescape-5495f8d47-7zsdx         1/1     Running            0             111s
kubevuln-54c4b4bf5f-bt7sl         1/1     Running            0             80s
node-agent-hnfn9                  1/1     Running            0             107s
operator-66b6b77b6b-7lg2w         1/1     Running            0             110s
otel-collector-7fdbcfdcc5-4bht4   1/1     Running            0             110s
storage-854c6b64bf-72pz7          1/1     Running            0             41m

kubectl logs pods/gateway-76c58b6569-lt6pd -n kubescape
{"level":"info","ts":"2023-11-06T06:18:48Z","msg":"Image version","release":""}
{"level":"info","ts":"2023-11-06T06:18:48Z","msg":"loaded gw url (service discovery)","url":"wss://ens.euprod1.cyberarmorsoft.com"}
{"level":"info","ts":"2023-11-06T06:19:02Z","msg":"accepting websocket connection","url query":"clusterComponent=InClusterTriggerHandler&clusterName=kind-kubescape&customerGUID=xxx","id":9043739233290585946,"number of incoming websockets":1}
{"level":"info","ts":"2023-11-06T06:19:02Z","msg":"connecting to master","url":"wss://ens.euprod1.cyberarmorsoft.com/v1/waitfornotification?customerGUID=xxx"}
{"level":"info","ts":"2023-11-06T06:19:02Z","msg":"loaded credentials"}
{"level":"warn","ts":"2023-11-06T06:19:03Z","msg":"dialing websocket","attempt":1,"error":"failed dialing to: 'wss://ens.euprod1.cyberarmorsoft.com/v1/waitfornotification?customerGUID=xxx', reason: 'EOF'"}
{"level":"warn","ts":"2023-11-06T06:19:09Z","msg":"dialing websocket","attempt":2,"error":"failed dialing to: 'wss://ens.euprod1.cyberarmorsoft.com/v1/waitfornotification?customerGUID=xxx', reason: 'EOF'"}
{"level":"fatal","ts":"2023-11-06T06:19:15Z","msg":"failed to connect to master","url":"wss://ens.euprod1.cyberarmorsoft.com/v1/waitfornotification?customerGUID=xxx","error":"failed dialing to: 'wss://ens.euprod1.cyberarmorsoft.com/v1/waitfornotification?customerGUID=xxx', reason: 'EOF'"}

Not sure if there is any incoming http request from kubescape that is causing the gateway pod to fail ? The firewall on my side will block incoming requests and it will not reach my kubernetes control plane. The VM I'm running my kind cluster does not have a public IP

from kubescape.

@extensure123 The gateway opens a websocket connection (which is bi-directional) to the following endpoint wss://ens.euprod1.cyberarmorsoft.com as you can see in the logs; so it sounds like this will have to be opened for incoming requests.

from kubescape.

@extensure123 any news regarding your issue?

from kubescape.

@amirmalka - Appreciate your inputs. Unfortunately, the firewall policies at my end will not allow any incoming connection without a long process approval. As the wss://ens.euprod1.cyberarmorsoft.com endpoint seems to be blocked by network team for my on-prem setup, I'm closing this out and explore this on a cloud VM where I have more control with ACL and endpoints.
Thanks all for the feedback and support.

from kubescape.

@amirmalka - Appreciate your inputs. Unfortunately, the firewall policies at my end will not allow any incoming connection without a long process approval. As the wss://ens.euprod1.cyberarmorsoft.com endpoint seems to be blocked by network team for my on-prem setup, I'm closing this out and explore this on a cloud VM where I have more control with ACL and endpoints. Thanks all for the feedback and support.

There are ways to run our commercial product on-prem, we are not allowed to advertise it here but you can reach out by email.

from kubescape.