josepdcs / kubectl-prof Goto Github PK

kubectl-prof is a kubectl plugin to profile applications on kubernetes with minimum overhead

License: Apache License 2.0

Makefile 2.11% Go 90.53% Python 2.46% Shell 2.88% Dockerfile 1.37% C 0.13% Java 0.19% JavaScript 0.14% Ruby 0.09% Rust 0.10%

profiling kubernetes java ebpf golang nodejs perf python clang

kubectl-prof's Introduction

Kubectl Prof

This is a kubectl plugin that allows you to profile applications with low-overhead in Kubernetes environments by generating FlameGraphs and many other outputs as JFR, thread dump, heap dump and class histogram for Java applications by using jcmd. For Python applications, thread dump output and speed scope format file are also supported. See Usage section. More functionalities will be added in the future.

Running kubectl-prof does not require any modification to existing pods.

This is an open source fork of kubectl-flame with several new features and bug fixes.

Requirements
Usage
Installation
How it works
Contribute
Maintainers
License

Requirements

Supported languages: Go, Java (any JVM based language), Python, Ruby, NodeJS, Clang and Clang++.
Kubernetes that use some of the following container runtimes:
- Containerd by using flag --runtime=containerd (default)
- CRI-O by using flag --runtime=crio

Usage

Profiling Java Pod

In order to profile a Java application in pod mypod for 1 minute and save the flamegraph into /tmp run:

kubectl prof my-pod -t 5m -l java -o flamegraph --local-path=/tmp

NOTICE:

if --local-path is omitted, flamegraph result will be saved into current directory

Profiling Alpine based container

Profiling Java application in alpine based containers require using --alpine flag:

kubectl prof mypod -t 1m --lang java -o flamegraph --alpine

NOTICE: this is only required for Java apps, the --alpine flag is unnecessary for other languages.

Profiling Java Pod and generate JFR output by using `jcmd` as default tool

Profiling Java Pod and generate JFR output require using -o/--output jfr option:

kubectl prof mypod -t 5m -l java -o jfr

Profiling Java Pod and generate JFR output but by using `async-profiler`

In this case, profiling Java Pod and generate JFR output require using -o/--output jfr and --tool async-profiler options:

kubectl prof mypod -t 5m -l java -o jfr --tool jcmd

Profiling Java Pod and generate thread dump output by using `jcmd` as default tool

In this case, profiling Java Pod and generate the thread dump output require using -o/--output threaddump options:

kubectl prof mypod -l java -o threaddump

Profiling Java Pod and generate heap dump output (hprof format) by using `jcmd` as default tool

In this case, profiling Java Pod and generate the heap dump output require using -o/--output heapdump options:

kubectl prof mypod -l java -o heapdump --tool jcmd

Profiling Java Pod and generate heap histogram output (hprof format) by using `jcmd` as default tool

In this case, profiling Java Pod and generate the heap histogram output require using -o/--output heaphistogram options:

kubectl prof mypod -l java -o heaphistogram --tool jcmd

Profiling specifying the container runtime

Supported container runtimes values are: crio, containerd.

kubectl prof mypod -t 1m --lang java --runtime crio

Profiling Python Pod

In order to profile a Python application in pod mypod for 1 minute and save the flamegraph into /tmp run:

kubectl prof mypod -t 1m --lang python -o flamegraph --local-path=/tmp

Profiling Python Pod and generate thread dump output

In this case, profiling Python Pod and generate the thread dump output require using -o/--output threaddump option:

kubectl prof mypod -t 1m --lang python --local-path=/tmp -o threaddump

Profiling Python Pod and generate speed scope output format file

In this case, profiling Python Pod and generate the thread dump output require using -o/--output speedscope option:

kubectl prof mypod -t 1m --lang python --local-path=/tmp -o speedscope

Profiling Golang Pod

In order to profile a Golang application in pod mypod for 1 minute run:

kubectl prof mypod -t 1m --lang go -o flamegraph

Profiling Node Pod

In order to profile a Python application in pod mypod for 1 minute run:

kubectl prof mypod -t 1m --lang node -o flamegraph

Profiling Ruby Pod

In order to profile a Ruby application in pod mypod for 1 minute run:

kubectl prof mypod -t 1m --lang ruby -o flamegraph

Profiling Clang Pod

In order to profile a Clang application in pod mypod for 1 minute run:

kubectl prof mypod -t 1m --lang clang -o flamegraph

Profiling Clang++ Pod

In order to profile a Clang++ application in pod mypod for 1 minute run:

kubectl prof mypod -t 1m --lang clang++ -o flamegraph

Profiling with several options:

Profiling a pod for 5 minutes in intervals of 60 seconds for java language by giving the cpu limits, the container runtime, the agent image and the image pull policy

kubectl prof mypod -l java -o flamegraph -t 5m --interval 60s --cpu-limits=1 -r containerd --image=localhost/my-agent-image-jvm:latest --image-pull-policy=IfNotPresent

Profiling in contprof namespace a pod running in contprof-apps namespace by using the profiler service account for go language

kubectl prof mypod -n contprof --service-account=profiler --target-namespace=contprof-stupid-apps -l go

Profiling by setting custom resource requests and limits for the agent pod (default: neither requests nor limits are set) for python language

kubectl prof mypod --cpu-requests 100m --cpu-limits 200m --mem-requests 100Mi --mem-limits 200Mi -l python

For more detailed options run:

kubectl prof --help

Installation

Using krew

Install Krew

Install repository and plugin:

kubectl krew index add kubectl-prof https://github.com/josepdcs/kubectl-prof
kubectl krew search kubectl-prof
kubectl krew install kubectl-prof/prof
kubectl prof --help

Pre-built binaries

See the release page for the full list of pre-built assets. And download the binary according yours architecture.

Installing for Linux x86_64

curl -sL https://github.com/josepdcs/kubectl-prof/releases/download/1.2.3/kubectl-prof_1.2.3_linux_x86_64.tar.gz
tar xvfz kubectl-prof_1.2.3_linux_x86_64.tar.gz && sudo install kubectl-prof /usr/local/bin/

Building

Install source code and golang dependencies

$ go get -d github.com/josepdcs/kubectl-prof
$ cd $GOPATH/src/github.com/josepdcs/kubectl-prof
$ make install-deps

Build binary

$ make

Build Agents Containers

Modify Makefile, property DOCKER_BASE_IMAGE, and run:

$ make agents

How it works

kubectl-prof launch a Kubernetes Job on the same node as the target pod. Under the hood kubectl-profcan use the following tools according the programming language:

For Java:
- async-profiler in order to generate flame graphs or JFR files and the rest of output type supported for this tool.
  - For generating flame graphs use the option: --tool async-profiler and -o flamegraph.
  - For generating JFR files use the option: --tool async-profiler and -o jfr.
  - For generating collapsed/raw use the option: --tool async-profiler and -o collapsed or -o raw.
  - Note: Default output is flame graphs if no option -o/--output is given.
- jcmd in order to generate: JFR files, thread dumps, heap dumps and heap histogram.
  - For generating JFR files use the options: --tool jcmd and -o jfr.
  - For generating thread dumps use the options: --tool jcmd and -o threaddump.
  - For generating heap dumps use the options: --tool jcmd and -o heapdump.
  - For generating heap histogram use the options: --tool jcmd and -o histogram.
  - Note: Default output is JFR if no option -o/--output is given.
- Note: Default tool is async-profiler if no option --tool is given and default output is flame graphs if no option -o/--output is also given.
For Golang: ebpf profiling.
- For generating flame graphs use the option: -o flamegraph.
- For generating raw use the option: -o raw.
- Note: Default output is flame graphs if no option -o/--output is given.
For Python: py-spy.
- For generating flame graphs use the option: -o flamegraph.
- For generating thread dumps use the option: -o threaddump.
- For generating speed scope use the option : -o speedscope.
- For generating raw use the option: -o raw.
- Note: Default output is flame graphs if no option -o/--output is given.
For Ruby: rbspy.
- For generating flame graphs use the option: -o flamegraph.
- For generating speed scope use the option : -o speedscope.
- For generating callgrind use the option: -o callgrind.
- Note: Default output is flame graphs if no option -o/--output is given.
For Node.js: ebpf profiling and perf but last one is not recommended.
- For generating flame graphs use the option: -o flamegraph.
- For generating raw use the option: -o raw.
- Note: Default output is flame graphs if no option -o/--output is given.
- In order for Javascript Symbols to be resolved, node process needs to be run with --prof-basic-prof flag.
For Clang and Clang++: perf is the default profiler but ebpf profiling is also supported.

The raw output is a text file with the raw data from the profiler. It could be used to generate flame graphs, or you can use https://www.speedscope.app/ to visualize the data.

kubectl-prof also supports to work in modes discrete and continuous:

In discrete mode: only one profiling result is requested. Once this result is obtained, the profiling process finishes. This is the default behaviour when only using -t time option.
In continuous mode: can produce more than one result. Given a session duration and an interval, a result is produced every interval until the profiling session finishes. Only the last produced result is available. It is client responsibility to store all the session results.
- For using this option you must use the --interval time option in addition to -t time.

In addition, kubectl-prof will attempt to profile all the processes detected in the container. It will try to profile them all based on the provided language. When this happens, the tool will display a warning similar to:

⚠ Detected more than one PID to profile: [2508 2509]. It will be attempt to profile all of them. Use the --pid flag specifying the corresponding PID if you only want to profile one of them.

But if you want to profile a specific process, you have two options:

Provide the specific PID using the --pid PID flag if you know the PID (the previous warning can help you identify the PID you want to profile).
Provide a process name using the --pgrep process-matching-name flag.

Contribute

Please refer to the contributing.md file for information about how to get involved. We welcome issues, questions, and pull requests

Maintainers

Josep Damià Carbonell Seguí: [email protected]

Special thanks to the original Author of kubectl-flame

Eden Federman: [email protected]
Verizon Media Code

License

This project is licensed under the terms of the Apache 2.0 open source license. Please refer to LICENSE for the full terms.

Project status

Service	Status
Github Actions
GoReport

kubectl-prof's People

Contributors

Stargazers

Watchers

Forkers

josdam sattishv slach

kubectl-prof's Issues

--runtime-path appears not to work

Hello @josepdcs. Thank you for your effort on developing this tool.

I noticed something that there is an option --runtime-path where one can specify an alternative container runtime install path.

root@kubectl-flame-658d9ffb4-zwfls:/workspace# kubectl prof | grep runtime-path
      --runtime-path string             Use a different container runtime install path (default "/run/containerd/")
...

However, no matter what I set this to, like /host/var/snap/microk8s/common/run/containerd/ or /host/data/snap/microk8s/common/run/containerd/, kubectl-prof appears to ignore this.

{"type":"log","data":{"time":"2024-01-16T05:13:36.376271942Z","level":"debug","msg":"{\"Duration\":5000000000,\"Interval\":5000000000,\"UID\":\"21488509-a797-4dc9-b3db-ff368fa1c55a\",\"ContainerRuntime\":\"containerd\",\"ContainerID\":\"804b035d095a1843d1baedff11f9e38f9f9cd967731e458f65feb15d1de6e9fe\",\"PodUID\":\"db24dd86-715f-4de3-9f37-962aa5aa32b7\",\"Language\":\"java\",\"Event\":\"itimer\",\"Compressor\":\"gzip\",\"Tool\":\"async-profiler\",\"OutputType\":\"flamegraph\",\"FileName\":\"\",\"HeapDumpSplitInChunkSize\":\"\",\"AdditionalArguments\":null}"}} 2024-01-16T05:13:36.376801051Z {"type":"progress","data":{"time":"2024-01-16T05:13:36.376699686Z","stage":"started"}} 2024-01-16T05:13:36.376804395Z {"type":"log","data":{"time":"2024-01-16T05:13:36.376769131Z","level":"debug","msg":"The target filesystem is: /run/containerd/io.containerd.runtime.v2.task/k8s.io/804b035d095a1843d1baedff11f9e38f9f9cd967731e458f65feb15d1de6e9fe/rootfs"}} 2024-01-16T05:13:36.378534835Z {"type":"error","data":{"reason":"read file failed: /run/containerd/io.containerd.runtime.v2.task/k8s.io/804b035d095a1843d1baedff11f9e38f9f9cd967731e458f65feb15d1de6e9fe/init.pid: open /run/containerd/io.containerd.runtime.v2.task/k8s.io/804b035d095a1843d1baedff11f9e38f9f9cd967731e458f65feb15d1de6e9fe/init.pid: no such file or directory"}}
{"type":"log","data":{"time":"2024-01-16T05:18:36.378708007Z","level":"warn","msg":"Maximum allowed time 5m0s surpassed. Cleaning up and auto-deleting the agent..."}} 2024-01-16T05:18:36.378825029Z {"type":"log","data":{"time":"2024-01-16T05:18:36.378739311Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh stop"}}

As one can see in /run/containerd/io.containerd.runtime.v2.task/k8s.io/804b035d095a1843d1baedff11f9e38f9f9cd967731e458f65feb15d1de6e9fe/init.pid: no such file or directory", it is always /run/containerd/ that it attempts to read from.

This issue is preventing my from profiling pods running on a microk8s based node, with containerd apparently installed by snap. Is there any workaround available for my situation?

Thank you.

Provide the RAW output format

The raw output format is provided for the most profiling tools (async-profiler, py-spy, rbspy, perf, bcc-profile, etc.) and it can be transformed to framegraph by using Brendan Greed's tool or, even, and this is most worth, it can be read by SpeedScope tool

Set containerd as default container runtime over crio

Support for specify process name

First of all, @josepdcs thank you for your efforts, this project is the best supported project for performance sampling so far.

When a application container that contains multiple processes require specifying the target process name when capturing. Just like the --pgrep parameter inside the upstream

Support for specifying the concrete PID

Could be useful being able to specify the concrete PID to be profiled if it is known --pid 343434

Clang++ target attempts to create profiler pod with invalid name

% kubectl prof normcore-room-07156def-a20e-411e-8c4e-8775f63e0d0b -t 1m --lang clang++ -o flamegraph
Default profiling tool bpf will be used ... ✔
Verifying target pod ... ✔
Launching profiler ... ❌
FATA[2024-02-15T15:50:04-05:00] Job.batch "kubectl-prof-clang++-bpf-a46bf2fc-a284-435f-87fe-34f3383959d1" is invalid: [metadata.name: Invalid value: "kubectl-prof-clang++-bpf-a46bf2fc-a284-435f-87fe-34f3383959d1": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.labels: Invalid value: "kubectl-prof-clang++-bpf-a46bf2fc-a284-435f-87fe-34f3383959d1": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]

It looks like the clang++ in the profiler pod name is failing on my kubernetes deployment. Is there a way to set a custom name or not include the language in the profiler pod name?

Implement a default configuration file

The default configuration could be stored in .kube folder as yaml file: kubectl-prof.yml

Krew action install does not work

Bump to Async-Profiler 3.0

New awesome async-profiler version: https://github.com/async-profiler/async-profiler/releases/tag/v3.0

capabilities in JobConfig can be reduced from SYS_ADMIN

for example in
https://github.com/josepdcs/kubectl-prof/blob/main/internal/cli/kubernetes/job/jvm.go#L76

The capability in the JobConfig for perf sampling can be lowered from SYS_ADMIN to just PERFMON and SYSLOG
kernels prior to v5.9 may require SYS_PTRACE

https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html#perf-events-access-control

The permissions required for perf are
sysctl -w kernel.kptr_restrict=0
sysctl -w kernel.perf_event_paranoid=1

or capabilities PERFMON and SYSLOG which is confirmed in the kernel code at the following locations
https://elixir.bootlin.com/linux/v5.15.148/source/tools/perf/util/util.c#L290
https://elixir.bootlin.com/linux/v5.15.148/source/kernel/kallsyms.c#L794

I modified the line mentioned, built and tested the plugin with java and async-profiler and the profiler returns the output. this is some of the output of --dry-run

      name: kubectl-prof
        resources:
          limits:
            cpu: "1"
        securityContext:
          capabilities:
            add:
            - PERFMON
            - SYSLOG
          privileged: true

Support for Arm64 architectures

Hi there,

Just wanted to know if there would be an interest for the support of Arm64 architectures ?
Any work launched on the topic or foreseen challenges ?

I can try to contribute if needed.

Thx !

PHP profiling

Here a low-overhead sampling profiler for PHP 7+

https://github.com/adsr/phpspy
https://github.com/reliforp/reli-prof

Support for applications running subprocesses

Being able to profile apps that launch more than one child process, even in model of hierarchical processes

It occurs in python, node.js, etc.

Rust profiling

Enable use of perf, bpf and https://github.com/flamegraph-rs/flamegraph

1.2.2 flamegraphs for java with -e alloc and --interval don't download (Checksum does not match) error

When I run with java async-profiler cpu events - all the flamegraphs are downloaded

--tool async-profiler -e cpu -l java -o flamegraph -t 2m --interval 60s

but with java async-profiler alloc events I get an error saying Checksum does not match

--tool async-profiler -e alloc -l java -o flamegraph -t 2m --interval 60s

Default profiling tool async-profiler will be used ... ✔
Verifying target pod ... ✔
Launching profiler ... ✔
Profiling ... ✔

Checksum does not match, retrying: /tmp/agent-flamegraph-1382909.html.gz...
Checksum does not match, retrying: /tmp/agent-flamegraph-1382909.html.gz...

with alloc events, when I remove --interval 60s a single flamegraph is produced and downloaded.

looks like a timing issue, I tried changing (perfDelayBetweenJobs) but it doesnt seem to introduce a delay at all as there is no delay between the jobs timestamps. I see the sleep in the code in ./internal/agent/profiler/jvm/async_profiler.go

{"type":"log","data":{"time":"2024-02-01T19:55:57.560460874Z","level":"debug","msg":"The target filesystem is: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9207e89a6d90b33dc9082d185a43c82a20c217e5b42bf42dd1dc33409829e9dc/rootfs"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.560837314Z","level":"debug","msg":"pgrep -P 3327888"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.580797179Z","level":"debug","msg":"pgrep -P 3328340"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.598762725Z","level":"debug","msg":"pgrep -P 3328373"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.615486974Z","level":"debug","msg":"The PIDs to be profiled: [3328373]"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.615520289Z","level":"debug","msg":"cp -r /app/async-profiler /tmp"}}
{"type":"log","data":{"time":"2024-02-01T19:55:57.618155249Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 60 -f /tmp/agent-flamegraph-3328373.html -e alloc --fdtransfer 3328373"}}
{"type":"log","data":{"time":"2024-02-01T19:56:57.756881691Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"result","data":{"time":"2024-02-01T19:56:57.757942168Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-3328373.html.gz","file-size-in-bytes":28507,"checksum":"2ddf9c65b8b12630eabfc2ffd6d9d61f","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-01T19:56:57.758313068Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 60 -f /tmp/agent-flamegraph-3328373.html -e alloc --fdtransfer 3328373"}}
{"type":"log","data":{"time":"2024-02-01T19:57:57.859065758Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"result","data":{"time":"2024-02-01T19:57:57.860134251Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-3328373.html.gz","file-size-in-bytes":27931,"checksum":"385cf11bce4a902a073427e3b521f28b","compressor-type":"gzip"}}
{"type":"log","data":{"time":"2024-02-01T19:57:57.860312937Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh -o flamegraph -d 60 -f /tmp/agent-flamegraph-3328373.html -e alloc --fdtransfer 3328373"}}
{"type":"log","data":{"time":"2024-02-01T19:58:57.961279147Z","level":"debug","msg":"stat -c%s /tmp/agent-flamegraph-3328373.html.gz"}}
{"type":"result","data":{"time":"2024-02-01T19:58:57.962588016Z","result-type":"flamegraph","file":"/tmp/agent-flamegraph-3328373.html.gz","file-size-in-bytes":34371,"checksum":"6673e8b7ae742ee567d29eb6dcc7d77c","compressor-type":"gzip"}}
{"type":"progress","data":{"time":"2024-02-01T19:58:57.962825035Z","stage":"ended"}}
{"type":"log","data":{"time":"2024-02-01T20:03:57.976570146Z","level":"warn","msg":"Maximum allowed time 5m0s surpassed. Cleaning up and auto-deleting the agent..."}}
{"type":"log","data":{"time":"2024-02-01T20:03:57.976643458Z","level":"debug","msg":"/tmp/async-profiler/profiler.sh stop 3328373"}}
{"type":"log","data":{"time":"2024-02-01T20:03:57.990292613Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-3328373.html"}}
{"type":"log","data":{"time":"2024-02-01T20:03:57.990424459Z","level":"debug","msg":"Trying to remove file: /tmp/agent-flamegraph-3328373.html.gz"}}

maybe if the filename had a timestamp or an interval counter rather than the same file filename agent-flamegraph-1382909.html for each interval, it would help. Also the perfRecordOutputFileName and perfScriptOutputFileName ?

Complete Readme

Use CO-RE to remove dependency on host machine kernel headers?

Hey all, I've been able to get kubectl-prof working with the perf profiler and it works beautifully (thanks for the hard work here!). However, I'd really like to use the eBPF profiler, but on DigitalOcean, their host instances do not have the kheaders module available.

I've reached out to DO and it doesn't seem like the lack of kheaders is going to change anytime soon. I was looking at CO-RE (https://web.archive.org/web/20220522105208/https://www.seekret.io/blog/handling-the-challenge-of-deploying-ebpf-into-the-wild/) as a potential work around.

Is this something that I could help implement here?