groundcover-com / caretta Goto Github PK

Modified caretta agent that can run on pre-4.16 linux kernels and optionally uses os ip resolver (not k8s). You can see it here: https://github.com/gelonsoft/caretta-nonebpf
I'm not professional programmer, so code is very very bad, but looks like it works)

Is your feature request related to a problem? Please describe.

1. Run caretta agent on pre-4.16 linux kernels. To get connection list we can use standard OS interface like netstat do
2. Run caretta agent on windows using standard OS interface like netstat do
3. Run on non-k8s machines like bare metal servers or general VM. To resolve IP addresses it uses general OS reverse DNS resolve

Describe the solution you'd like

1. Optionally use netstat-like OS interfaces to get connections list instead of ebpf
2. Optionally use netstat-like OS interfaces to run on windows VM
3. Optionally use OS ip dns reverse IP resolve instead of k8s to make it run on non-k8s environments

Describe alternatives you've considered
Check my modified agent here: https://github.com/gelonsoft/caretta-nonebpf

Additional context
Modified agent uses github.com/cakturk/go-netstat package to get connections list using OS standard interfaces. This package support windows and linux OS.

We have configured the caretta on our development EKS 1.24 . we ran into some error,

logger=context userId=0 orgId=1 uname= t=2023-10-24T19:09:17.663253879Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Get \"http://caretta-vm:8428/api/v1/query?query=topk%287%2C+sum+by+%28client_name%2C+server_name%29+%28+rate%28+%28caretta_links_observed%7Bclient_namespace%3D~%22%28.%2A%29%22%2C+client_kind%3D~%22%28.%2A%29%22%2C+client_name%3D~%22%28.%2A%29%22%2C+server_port%3D~%22%28.%2A%29%22%2C+client_kind%21~%22%28node%7Cexternal%29%22%2C%7D+or+caretta_links_observed%7Bserver_namespace%3D~%22%28.%2A%29%22%2C+server_kind%3D~%22%28.%2A%29%22%2C+server_name%3D~%22%28.%2A%29%22%2C+server_port%3D~%22%28.%2A%29%22%2C+server_kind%21~%22%28node%7Cexternal%29%22%7D%29%5B300s%3A15s%5D%29+%29+%29&time=1698174540\": dial tcp 100.67.86.25:8428: i/o timeout" remote_addr=127.0.0.1 traceID= logger=context userId=0 orgId=1 uname= t=2023-10-24T19:09:17.66332854Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=127.0.0.1 time_ms=10002 duration=10.002311772s size=116 referer="http://localhost:58602/?orgId=1" handler=/api/ds/query logger=context userId=0 orgId=1 uname= t=2023-10-24T19:09:17.664494229Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Get \"http://caretta-vm:8428/api/v1/query_range?end=1698174540&query=sum+by+%28server_port%29+%28increase%28%28caretta_links_observed%7Bclient_namespace%3D~%22%28.%2A%29%22%2C+client_kind%3D~%22%28.%2A%29%22%2C+client_name%3D~%22%28.%2A%29%22%2C+server_port%3D~%22%28.%2A%29%22%7D+or+caretta_links_observed%7Bserver_namespace%3D~%22%28.%2A%29%22%2C+server_kind%3D~%22%28.%2A%29%22%2C+server_name%3D~%22%28.%2A%29%22%2C+server_port%3D~%22%28.%2A%29%22%7D%29%5B300s%3A15s%5D%29%29+%3E+0&start=1698174240&step=15\": dial tcp 100.67.86.25:8428: i/o timeout" remote_addr=127.0.0.1 traceID= logger=context userId=0 orgId=1 uname= t=2023-10-24T19:09:17.66457895Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=127.0.0.1 time_ms=10009 duration=10.009525523s size=116 referer="http://localhost:58602/?orgId=1" handler=/api/ds/query logger=context userId=0 orgId=1 uname= t=2023-10-24T19:09:54.37720622Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=127.0.0.1 time_ms=0 duration=854.397µs size=29 referer="http://localhost:58602/?orgId=1" handler=/ logger=http.server t=2023-10-24T19:10:00.467825731Z level=info msg="Successful Login" User=admin@localhost Logs from 21/10/2023, 13:44:29

Hi,

I read your article in https://medium.com/better-programming/improve-cluster-monitoring-with-network-mapping-in-grafana-fa8bb479fd47

Thank you for that. I tried Caretta on my hybrid home cluster that has hybrid architecture amd64 and arm64 and I see that you don't provide multi architecture images that run on both architectures.

So please add arm64 support.

Describe the bug
Caretta pod don't to runing, please see follow error prompt.
2023/06/12 03:10:59 Caretta starting...
2023/06/12 03:10:59 Error watching cluster's state:
To Reproduce
Steps to reproduce the behavior:

1. download release version for chart link https://github.com/groundcover-com/caretta/archive/refs/tags/v0.0.16.zip
2. helm install caretta . --namespace caretta

Expected behavior
All pod can be to normal running status
Screenshots

Environment (please complete the following information):

• OS: Centos7.9 5.4.210-1.el7.elrepo.x86_64
• Browser chrome
• Kubernetes cluster information - distribution, version: v1.18.20
• Caretta version: v0.0.16

Additional context

Is your feature request related to a problem? Please describe.
I would like to sample caretta across some nodes I have running on kube v1.24.6, however, I currently cannot deploy as Caretta wants to use PodSecurityPolicy. This has been deprecated in Kube versions 1.21 and has been removed 1.25. Kubernetes recommend migrating from the PodSecurityPolicy to the built in PodSecurity Admission Controller - here

Describe the solution you'd like
Groundcover to apply fix by enabling support for kube versions 1.21 and greater

Describe alternatives you've considered
No alternatives

Additional context

the server could not find the requested resource

i dont why

Is your feature request related to a problem? Please describe.

The main caretta pod cannot be started with the following error:
2023/03/26 06:45:23 Warning: couldn't retrieve the owner of docker-desktop - Unsupported kind for lookup - Node. This might happen when starting up
2023/03/26 06:45:33 Couldn't load probes - error loading BPF objects from go-side. field HandleSockSetState: program handle_sock_set_state: apply CO-RE relocations: load kernel spec: no BTF found for kernel version 5.15.49-linuxkit: not supported

Describe the solution you'd like
A clear and concise description of what you want to happen.
I'd like caretta to run under Docker-Dekstop Kubernetes in MacOS
My HW:
CPU - Apple M1 Max
Arch: ARM64
OS: MacOS Monterey 12.6.3

Describe alternatives you've considered
Kind support, cloud integration, I am testing a lot of stuff in my local environment and I'd like caretta to run under the same e environment, first to test it and then use it of course.

Additional context
This project looks awesome, introduced by company's Dev team in KDC2023.

Thanks

Hi, our k8s PaaS names our deployments like kd-111205-5e90c9fa-e39feabb-deployment and inside the manifests adds metadata.annotations and/or metadata.labels that give the sense of the application deployed.

Could be possible to set a custom applicaction name composed by that info?

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "563"
    kpaas/component.domain: test.com
    kpaas/component.name: app
    kpaas/component.version: 9.1.11
    kpaas/deployment.domain: devel
    kpaas/deployment.name: gapp
    kpaas/owner: devel
    kpaas/role: app00
    kpaas/service.domain: test.com
    kpaas/service.name: app
    kpaas/service.version: 30.0.0
    kpaas/translations: '{"k-2f6e1093":"app","k-700b652f":"prometheuscs","k-7c3044cd":"appcs"}'
  creationTimestamp: "2023-02-20T12:51:00Z"
  generation: 589
  labels:
    kpaas/component.domain: a0262a2b
    kpaas/component.name: 2f6e1093
    kpaas/component.version: a41e6ca7
    kpaas/controller: deploymentcontroller
    kpaas/deployment.domain: 2d93289d
    kpaas/deployment.id: kd-125052-7a101811
    kpaas/deployment.name: f209fbf9
    kpaas/owner: 2d93289d
    kpaas/role: cb30c12b
    kpaas/service.domain: a0262a2b
    kpaas/service.name: d5430443
    kpaas/service.version: 4f9e51e4
...

Same here coroot/coroot#135

Thanks

Is your feature request related to a problem? Please describe.
I am very happy to be able to look at the Caretta so that we can clearly see the application topology in the k8s cluster.
Through Caretta, we can see the topological relationship between applications in detail, but the network packets between applications are not very detailed.

Could we consider introducing the following network detail indicators?

• Such as network equipment (inbound and outbound bandwidth, errors and discarded data packets, etc.) dev_egress_bytes、dev_ingress_bytes、dev_egress_pkts and so on.
• IP/UDP (inbound and outbound data packets, etc.) ip_ingress_drop_pkts、ip_egress_drop_pkts.
• TCP protocol (number of connections, retransmissions, etc.) tcp_conn、tcp_conn_latency、tcp_backlog、tcp_conn_cwnd.

We can clearly see the network connection of applications in k8s cluster by implementing above functions.

Describe the solution you'd like

Display the traffic topology diagram between applications by implementing the bpf program that monitors key data of network protocols.

Describe alternatives you've considered

Additional context

Describe the bug
Couldn't load probes - error loading BPF objects from go-side. field HandleSockSetState: program handle_sock_set_state: map sock_infos: map create: operation not permitted (MEMLOCK may be too low, consider rlimit.RemoveMemlock)

To Reproduce
Steps to reproduce the behavior:

1. Go to one of carettas failing pods
2. Click on logs
3. Scroll down to the last line
4. See error

Expected behavior
healthy pods but instead pods are restarting over and over again and failing
Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: linux
• Kubernetes cluster information - k8s v1.23

Additional context
same configuration works on aws clusters only failing in azure and gcp

Describe the bug
Missing daemonset, OOMKilled.

To Reproduce
helm install caretta -n caretta --create-namespace groundcover/caretta

Screenshots

aks environment

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME aks-nodepool1-0 Ready agent 186d v1.26.6 10.224.4.18 <none> Ubuntu 22.04.2 LTS 5.15.0-1042-azure containerd://1.7.1+azure-1 aks-nodepool1-1 Ready agent 186d v1.26.6 10.224.4.37 <none> Ubuntu 22.04.3 LTS 5.15.0-1042-azure containerd://1.7.1+azure-1 aks-nodepool1-2 Ready agent 186d v1.26.6 10.224.4.42 <none> Ubuntu 22.04.3 LTS 5.15.0-1042-azure containerd://1.7.1+azure-1 aks-nodepool1-3 Ready agent 186d v1.26.6 10.224.4.22 <none> Ubuntu 22.04.3 LTS 5.15.0-1042-azure containerd://1.7.1+azure-1 aks-nodepool1-4 Ready agent 186d v1.26.6 10.224.4.44 <none> Ubuntu 22.04.3 LTS 5.15.0-1042-azure containerd://1.7.1+azure-1 aks-nodepool1-5 Ready agent 186d v1.26.6 10.224.4.51 <none> Ubuntu 22.04.3 LTS 5.15.0-1042-azure containerd://1.7.1+azure-1 aks-nodepool1-6 Ready agent 186d v1.26.6 10.224.4.23 <none> Ubuntu 22.04.3 LTS 5.15.0-1042-azure containerd://1.7.1+azure-1 aks-nodepool1-7 Ready agent 186d v1.26.6 10.224.4.52 <none> Ubuntu 22.04.3 LTS 5.15.0-1042-azure containerd://1.7.1+azure-1

caretta environment

Chart Version: caretta-0.0.16
Caretta Version: v0.0.16

caretta log

2024/03/05 09:19:34 Warning: couldn't retrieve owner of webapp-54db647b5c - Missing replicaset for UID 8dfac9e4-0e7e-4f0d-951b-1c352619dcac. This might happen when starting up 2024/03/05 09:18:10 Warning: couldn't retrieve owner of main - Unsupported kind for lookup - Alertmanager. This might happen when starting up 2024/03/05 09:18:10 Warning: couldn't retrieve owner of k8s - Unsupported kind for lookup - Prometheus. This might happen when starting up 2024/03/05 09:18:10 Warning: couldn't retrieve owner of k8s - Unsupported kind for lookup - Prometheus. This might happen when starting up 2024/03/05 09:18:10 Warning: couldn't retrieve owner of main - Unsupported kind for lookup - Alertmanager. This might happen when starting up 2024/03/05 09:18:10 Warning: couldn't retrieve owner of main - Unsupported kind for lookup - Alertmanager. This might happen when starting up