When using Kubernetes nodes with RHEL 9, the `node-agent` pods fail to start about kubescape HOT 13 CLOSED

Yup, same kernel, same issue. I will update here when we have the fix...

from kubescape.

@ioannisgk can you post here your kernel version from uname -a or equivalent?

from kubescape.

Also, which Kubernetes installer did you use?
I am thinking it might come from Pod Security Admission: https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-namespace-labels/
I had similar issue on Talos: https://www.talos.dev/v1.6/kubernetes-guides/configuration/pod-security/

from kubescape.

@matthyx

Thank you for the information. This is a self-managed cluster in RHEL 9 nodes, where I installed Kubernetes with the kubeadm init command. The exact same setup works with Ubuntu 22.04, this problem only occurs on RHEL nodes.

Nodes OS: RHEL v9.3
Linux kernel version: Linux kubernetes-master1 5.14.0-362.18.1.el9_3.x86_64
Helm chart: v1.18.1
Kubernetes Server: v1.28.3
Kubernetes Client: v1.28.3
Calico version: v3.26.3
Containerd version: 1.6.27

from kubescape.

I have replicated your error, and currently checking with our ebpf provider: inspektor-gadget/inspektor-gadget#2444

from kubescape.

I have the same problem
time="2024-02-02T15:24:07Z" level=info msg="container-hook: monitoring runtime at /host/usr/local/bin/runc"
time="2024-02-02T15:24:07Z" level=info msg="container-hook: monitoring runtime at /host/usr/bin/crun"
time="2024-02-02T15:24:07Z" level=info msg="container-hook: monitoring runtime at /host/usr/bin/conmon"
{"level":"error","ts":"2024-02-02T15:24:08Z","msg":"error starting exec tracing","error":"creating tracer: attaching exit tracepoint: cannot create bpf perf link: permission denied"}
{"level":"fatal","ts":"2024-02-02T15:24:08Z","msg":"error starting the container watcher","error":"starting app behavior tracing: creating tracer: attaching exit tracepoint: cannot create bpf perf link: permission denied"}

i'm using rocky-linux / 5.14.0-362.13.1.el9_3.x86_64

from kubescape.

after further investigation, it is a bug in the RHEL kernel and the inspektor-gadget team has opened a PR for a fix: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3717

from kubescape.

@ioannisgk may I ask you to raise this issue to your Red Hat success manager to speed up the merge request above?

from kubescape.

I have the same issue but I'm using EKS managed nodes running bottlerocket.

I am deploying using the helm chart.

from kubescape.

@gabrielrinaldi I can confirm the issue is the same, but the reason is different. Can you open a new issue and we'll discuss the solutions there?

from kubescape.

@ioannisgk I have a workaround if you want: while the kernel version usually remains the same for the whole lifecycle of a RHEL release, there is a way to install a more recent one...

I have followed this page https://wiki.crowncloud.net/?Installing_the_Linux_Kernel_6x_on_AlmaLinux_9 and installed https://elrepo.org/tiki/kernel-lt (because I wanted the latest "long term support" branch.

I can confirm our ebpf capabiilities are working with this kernel:

$ uname -a
Linux localhost.localdomain 6.1.77-1.el9.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Feb  5 16:34:15 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

from kubescape.

@ioannisgk can I hear back from you?

from kubescape.

I updated the documentation with the proposed workaround.

from kubescape.