iovisor / kubectl-trace Goto Github PK

It looks like the image used by the tracer doesn't support GKE, as it doesn't have the necessary headers, I get the following output when I try to run the trace:

if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
chdir(/lib/modules/4.14.65+/build): No such file or directory
definitions.h:1:10: fatal error: 'net/sock.h' file not found
Unknown struct/union: 'sock'

Looking at the image build, it's not surprising that this is the result. @jasonkeene 's towel tool, which is quite a similar tool, solves the problem of fetching google's headers in this way:

https://github.com/jasonkeene/towel/blob/master/docker/download-chromium-os-kernel-source

Perhaps some logic could be added to check the host OS version, and if it is google's container os download the headers using a similar methodology?

A kubectl trace logs command would be useful for those use cases where is preferrable to not use a tty to get out information from a trace execution.

Examples:

kubectl trace logs -n mynamespace 4196deda-f0b5-11e8-93d3-8c164500a77e

kubectl trace logs 4196deda-f0b5-11e8-93d3-8c164500a77e

Status ad age are missing fields in the table printed out on a kubectl trace get

Those fields can be inferred by getting status and age from the related trace's job.

See: https://github.com/iovisor/bpftrace/blob/75a012c7e171b4acdf5a66469ec6be34e79dd4da/docs/reference_guide.md#2-kernel-headers-not-found

Original command in master to get the binary is to use curl with -Lo

curl -Lo https://github.com/iovisor/kubectl-trace/releases/download/v0.1.0-rc.1/kubectl-trace_0.1.0-rc.1_linux_amd64.tar.gz

curl -LO will be more properly.

Error Log

root@node1:~# curl -Lo https://github.com/iovisor/kubectl-trace/releases/download/v0.1.0-rc.1/kubectl-trace_0.1.0-rc.1_linux_amd64.tar.gz
curl: no URL specified!
curl: try 'curl --help' or 'curl --manual' for more information

Fixed

root@node1:~# curl -LO https://github.com/iovisor/kubectl-trace/releases/download/v0.1.0-rc.1/kubectl-trace_0.1.0-rc.1_linux_amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   634    0   634    0     0    817      0 --:--:-- --:--:-- --:--:--   815
  1 28.2M    1  509k    0     0   186k      0  0:02:35  0:00:02  0:02:33  497k

Env.

root@node1:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic


curl --version
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24

We don't want to allow tracejobs to accumulate without bound or accidentally left running, sapping system resources for traces no one will look at.

To achieve this, I think that setting activeDeadlineSeconds on the trace job would provide a built-in way to ensure that no job runs longer than a sane default (say, 1 hour), unless specifically requested with a command-line flag to override the deadline (eg, --deadline 24h) to extend the time that a job is allowed to run.

Ideally, this would also signal bpftrace to dump its map / summarize the map data before exiting, and could also be used as a mechanism for having jobs run for a specific duration. Eg, sample for 10 minutes than exit.

HI there,

Thank you very much for making this project.
I tried it in a GKE cluster running with COS nodes and got an error after creating a trace using

kubectl trace run node-name -e 'tracepoint:syscalls:sys_enter_open { printf("%s %s\n", comm,
    str(args->filename)); }'

The container cannot run because of the following error:

Containers:
  kubectl-trace-c59249b4-e5b5-11e9-9f93-f2189825cd80:
    Container ID:  docker://abca6aed039bd81ef98f4ca205c19040ae808477666b3ddfae509767c5e56916
    Image:         quay.io/iovisor/kubectl-trace-bpftrace:latest
    Image ID:      docker-pullable://quay.io/iovisor/kubectl-trace-bpftrace@sha256:fe1683d6782f4ad7690955543c048296b43b6c6a5046ea345a277318c92482d1
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/timeout
      --preserve-status
      --signal
      INT
      3600
      /bin/trace-runner
      --program=/programs/program.bt
    State:          Terminated
      Reason:       ContainerCannotRun
      Message:      OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/bin/timeout\": stat /bin/timeout: no such file or directory": unknown
      Exit Code:    127
      Started:      Thu, 03 Oct 2019 17:14:15 +0900
      Finished:     Thu, 03 Oct 2019 17:14:15 +0900

I checked the /bin/timeout existence in the docker image used:

➜ docker run -it --entrypoint "stat" quay.io/iovisor/kubectl-trace-bpftrace /bin/timeout
stat: can't stat '/bin/timeout': No such file or directory

The PR related to the source code is #88 but there is no trace of adding the binary to the Dockerfile.

Could you check it please?

Cheers,

Raphael

$ kubectl trace get
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x1d02385]

goroutine 1 [running]:
github.com/iovisor/kubectl-trace/pkg/tracejob.(*TraceJobClient).GetJob(0xc00066fc90, 0x0, 0x0, 0x0, 0x1d00903, 0xc00000c3c0, 0x0, 0x0)
	/Users/dalehamel/workspace/kubectl-trace/pkg/tracejob/job.go:135 +0x375
github.com/iovisor/kubectl-trace/pkg/cmd.(*GetOptions).Run(0xc0005a3280, 0x21af8e0, 0xc000379b00)
	/Users/dalehamel/workspace/kubectl-trace/pkg/cmd/get.go:163 +0x274
github.com/iovisor/kubectl-trace/pkg/cmd.NewGetCommand.func2(0xc0003d4000, 0x2a8f058, 0x0, 0x0, 0x0, 0x0)
	/Users/dalehamel/workspace/kubectl-trace/pkg/cmd/get.go:88 +0x94
github.com/spf13/cobra.(*Command).execute(0xc0003d4000, 0x2a8f058, 0x0, 0x0, 0xc0003d4000, 0x2a8f058)
	/Users/dalehamel/go/pkg/mod/github.com/spf13/[email protected]/command.go:762 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0xc0002d7900, 0xc0000da000, 0x215ede0, 0xc0000e6000)
	/Users/dalehamel/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2ea
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/dalehamel/go/pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
	/Users/dalehamel/workspace/kubectl-trace/cmd/kubectl-trace/root.go:24 +0x10e

Appears to be this https://github.com/iovisor/kubectl-trace/blob/master/pkg/tracejob/job.go#L134 (note my local file is different line num)

It looks like if you try to list the trace jobs before their are any, reading this is trying to read invalid memory. We probably need to check that it's safe to navigate the status struct before trying to access its members.

Two options come to my mind right now.

• from CI
  1. test
  2. build
  3. tag on github
  4. attach release with artifacts (ie., binaries) to such tag
• from local environment using a makefile command
  • step 2 to 4 above

hey,

You guys write some really awesome stuff. Are there any plans for getting kubectl-trace to work with the set of tools from https://github.com/iovisor/bcc?

Thanks,
Jon

root@rpi4-1:/home/pi/GO/src/github.com/kubectl-trace# make
rm -f -R _output
rm -f -R dist
CGO_ENABLED=1 go build -ldflags '-X github.com/iovisor/kubectl-trace/pkg/version.buildTime=1574941576 -X github.com/iovisor/kubectl-trace/pkg/version.gitCommit=9251ebac23648762fc19a7213df2e117bd654f0f -X github.com/iovisor/kubectl-trace/pkg/cmd.ImageNameTag=quay.io/iovisor/kubectl-trace-bpftrace:9251ebac23648762fc19a7213df2e117bd654f0f -X github.com/iovisor/kubectl-trace/pkg/cmd.InitImageNameTag=quay.io/iovisor/kubectl-trace-init:9251ebac23648762fc19a7213df2e117bd654f0f' -o _output/bin/kubectl-trace ./cmd/kubectl-trace
# github.com/kubectl-trace/cmd/kubectl-trace
cmd/kubectl-trace/root.go:24:29: cannot use streams (type "github.com/kubectl-trace/vendor/k8s.io/cli-runtime/pkg/genericclioptions".IOStreams) as type "github.com/iovisor/kubectl-trace/vendor/k8s.io/cli-runtime/pkg/genericclioptions".IOStreams in argument to cmd.NewTraceCommand
make: *** [Makefile:35: _output/bin/kubectl-trace] Error 2
root@rpi4-1:/home/pi/GO/src/github.com/kubectl-trace#

when try to build with docker image on make image/build I got

...
Step 9/11 : RUN apt-get update && apt-get install -y bpftrace=${bpftraceversion} && rm -rf /var/lib/apt/lists/* && apt-get clean
---> Running in b5c140ed9829
Get:1 http://ports.ubuntu.com/ubuntu-ports eoan InRelease [255 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports eoan-updates InRelease [97.5 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports eoan-backports InRelease [88.8 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports eoan-security InRelease [97.5 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports eoan/restricted armhf Packages [10.5 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports eoan/multiverse armhf Packages [149 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports eoan/main armhf Packages [1229 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports eoan/universe armhf Packages [11.3 MB]
Reading package lists...
E: Release file for http://ports.ubuntu.com/ubuntu-ports/dists/eoan-updates/InRelease is not valid yet (invalid for another 3d 9h 38min 43s). Updates for this repository will not be applied.
E: Release file for http://ports.ubuntu.com/ubuntu-ports/dists/eoan-backports/InRelease is not valid yet (invalid for another 3d 9h 38min 53s). Updates for this repository will not be applied.
E: Release file for http://ports.ubuntu.com/ubuntu-ports/dists/eoan-security/InRelease is not valid yet (invalid for another 3d 9h 38min 36s). Updates for this repository will not be applied.
The command '/bin/sh -c apt-get update && apt-get install -y bpftrace=${bpftraceversion} && rm -rf /var/lib/apt/lists/* && apt-get clean' returned a non-zero code: 100
make: *** [Makefile:63: image/build] Error 100
root@rpi4-1:/home/pi/GO/src/github.com/kubectl-trace#

Hi, I'm the maintainer of https://github.com/GoogleContainerTools/krew, a plugin manager for kubectl.

I would love to bring kubectl trace into krew which would help installation instructions for this tool, and would be a great to have example on Krew index which already has over 20 plugins.

Krew is currently being donated to kubectl project through KEP process and already has around 20 plugins distributed via krew.

Happy to host this project on krew, and see https://github.com/GoogleContainerTools/krew/blob/master/docs/DEVELOPER_GUIDE.md if you're interested.

Similar to the describe for normal kubernetes resources but for traces.
Nota bene: remember that kubectl trace does not create a custom resource but leverages kubernetes resources to inspect the target cluster with bpftrace programs, so this command can be slightly different from kubectl describe even if pursuing the same goals.

Should aggregate the events for the aggregated resources we create to do a trace.

The status information we will already get out to implement this will be usable to replace the <missing> fields in the get commands here

Since flame graph supports bpftrace I propose to create a new feature making users of kubectl-trace able to obtain flame graphs.

Not sure yet whether such feature should be implemented from a flag or from a new command.

We'll investigate further cc @fntlnz.

In #39 @BondAnthony made possible to have automated releases using goreleaser.
I'm opening this issue to follow up some improvements on that area so that we don't forget what's left to do:

• Since we want to test it for a while before adding it to CI, we are now running goreleaser locally when tagging. The CI does the release build for every PR and merge but does not do the release process. It's manual in the sense that a human executes the command that does the automatic release not in the sense that a human has to do the whole release process. We want to have this 100% automated via travis, see the goreleaser doc.
• The goreleaser is not building the docker images right now, we want to do that. Also, since it is building the binaries, we want to use the built ones instead of rebuilding them again inside the docker images (I'm talking about 'tracerunner' here).
• The released binaries now use the bpftrace image tagged as latest. They must use the one with the same commit has that we built with Travis when that commit has was published.

Github actions is now available and its been enabled for the iovisor github org.

Someone with admin access to this repo can add the QUAY_TOKEN to it like what is been done with travis.

The major advantage of actions over travis that I see are:

• Travis.org seems to be getting aged-out for travis.com
• Github actions offers much more concurrency (up to 20 jobs), so we could test kubectl trace maybe against different versions of k8s, and distros?

WDYT @leodido @fntlnz

Once bpftrace/bpftrace#363 is merged we want to disable test building for bpftrace using -DBUILD_TESTING=OFF in Dockerfile.bpftracebase

Kubectl trace is becoming quite complex, it has a lot of possible options and commands combinations, also there are features that are not completely clear, like is the namespace parameter used to allocate the kubectl-trace container or it is used to filter the pod, so I think we need to have a documentation website.

The possible options I was considering are:

• GitHub pages
• Netlify

I was trying to use kubectl trace against a cluster using cri-o as a container runtime.

I'm having suspects this issue I'm hitting is due to some misconfiguration on the minikube side because I can't even to a basic kubectl exec but I'll post here to investigate a bit further.

Steps to reproduce

Create a kubernetes cluster using cri-o

minikube start --network-plugin=cni  --container-runtime=cri-o 

Run kubectl trace against that cluster, using -a to attach

kubectl trace run -e 'kprobe:do_sys_open { printf("%s: %s\n", comm, str(arg1)) }' minikube -a 
trace 803db987-1a4f-11e9-9aae-8c164500a77e created

At this point I expect it to just work but the attach does not return anything, It would be nice to see an error here too in this case.

Try to exec the created pod

I investigated a bit further trying to exec the pod, i don't seem to be able to.

kubectl exec -ti kubectl-trace-803db987-1a4f-11e9-9aae-8c164500a77e-mbbnr sh
error: unable to upgrade connection: 404 page not found

Logs are working

Logs are working so the trace program is being executed, is just the attach not working

kubectl trace logs kubectl-trace-26db601a-1a4f-11e9-86a6-8c164500a77e
systemd-journal: /proc/4346/comm
systemd-journal: /proc/4346/cmdline
systemd-journal: /proc/4346/status
systemd-journal: /proc/4346/sessionid
systemd-journal: /proc/4346/loginuid
systemd-journal: /proc/4346/cgroup
conmon: /var/log/pods/24c59238-1a4f-11e9-92aa-0800274036be/kubectl-trac
crio: /mnt/sda1/var/lib/containers/overlay-images/c1b5e63c0b56cab77ea
crio: /mnt/sda1/var/lib/containers/overlay-images/c1b5e63c0b56cab77ea
crio: /mnt/sda1/var/lib/containers/overlay-images/c1b5e63c0b56cab77ea

I'd like to propose a descriptive testing of the CLI testing.

Inspiration come from the Go project.

Eg.,

• https://github.com/golang/go/blob/master/src/cmd/go/testdata/script/run_hello.txt
• https://github.com/golang/go/blob/master/src/cmd/go/testdata/script/clean_testcache.txt

This would simplify the readability of tests, but also it would simplify their writing.

Related to #25.

I think the fastest way to add this would be via a command line switch, but that would be cumbersome to type out.

To override the builtin values more persistently, reading from an environment variable or configmap (though, not sure where that would be stored) could also work. A local config file (does kubectl standardize this in any way?) would probably be best long-term, as there's some other hardcoded values such us number of retries that could also be tweaked.

For developers, this would also be nice to be able to quickly test locally-built images (should the need arise), or test out new versions of bpftrace/bcc with a custom build.

This would also make it easy to A/B test containers for performance, or to work-around bugs such as in #52 for #51

Right now when you run a bpftrace program against a pod, that is only using the pod to determine the actual binary to use to export symbols by giving you the $container_pid variable.

A better pod support would involve having the possibility to filter only pod tasks so that the results are only relevant to the specific pod.

Upstream issue on bpftrace: bpftrace/bpftrace#329

Our cluster nodes are running CoreOS and upon trying to test this tool we found the following error:

Error: failed to start container "kubectl-trace-2056455a-4836-11ea-a279-9078419759fa": Error response from daemon: error while creating mount source path '/usr/src': mkdir /usr/src: read-only file system

In CoreOS the main filesystem is read-only. Does the tool need to read/write there? Is it possible to configure it as user to avoid it? I couldn't find mentions in the docs or other issues on how to tackle this.

tl;dr

We don't want to start bpftrace as a cli anymore, we want to start a server that
can parse bpftrace programs and move the whole cli experience to kubectl-trace.

Long version

While developing and maintaining kubectl-trace with our current user experience we are facing a set of different challenges,
but first, let's describe how the current experience looks like:

The current experience is that kubectl-trace, starts a job containing a container with the bpftrace binary in it,
then, a tool called the "tracerunnre" starts bpftrace with a set of parameters and a TTY connection. That TTY connection
is used to "stream" the stdout and stderr of bpftrace directly to the TTY of the terminal that started kubectl-trace or that
attached to a particolar trace.

The first challenge, had been to develop a stable TTY over http fow that can sustain the output coming
from bpftrace and serve it to the user via the kubectl.

The second challenge for us is to accuratelly manage errors. We are not managing them.

The third chalenge is that since day zero we wanted kubectl-trace to be able to connect to
multiple containers/hosts at once.

Because of those challenges, we've been looking forward a solution.
The most natural we came up with consists in moving the cli experience from bpftrace to kubectl-trace directly.
That means that the new "trace" user experience will be:

• The user starts a trace via kubectl-trace
• The local kubectl-trace parses the program and serializes it for execution
• The job is created with the serialized program
• Inside the job, instead of tracerunner we start a gRPC server exposing an API that contains the bpftrace functionalities
• kubectl-trace connects to that API and gets the traces back

While the "attach" user experience will be:

• The user attaches to a trace via kubectl-trace
• kubectl-trace just connects to the existing server and gets the trace back

By doing this we face a set of requirements that however can be achieved:

• The server is gRPC over TLS
• The connection between the kubectl-trace and the server is authenticated using the usual kubernetes auth mechanisms

On the other hand we obtain also a set of advantages:

• Since we have a gRPC server/client we can connect to multiple servers and have an API that supports that
• We will not need to connect to a remote tty anymore, we don't even need a tty
• By parsing and serializing the program at client side we can manage errors better, e.g. parsing errors are given right away without having to start the job and everything

As described in bpftrace/bpftrace#266, it appears that linking to musl instead of glibc is what causes sporadic crashes (deterministically) when loading certain scripts.

For instance, when I run kubectl-trace using tcpconnect.bt from bpftrace/tools, it works fine. But, if I try and load tcpaccept.bt, I get a crash every time:

libclang: crash detected during parsing: {
  'source_filename' : 'definitions.h'
  'command_line_args' : ['clang', '-I', '/bpftrace/include', '-nostdinc', '-isystem', '/virtual/lib/clang/include', '-I/lib/modules/4.14.65+/build/arch/x86/include', '-I/lib/modules/4.14.65+/build/arch/x86/include/generated/uapi', '-I/lib/modules/4.14.65+/build/arch/x86/include/generated', '-I/lib/modules/4.14.65+/build/include', '-I/lib/modules/4.14.65+/build/./arch/x86/include/uapi', '-I/lib/modules/4.14.65+/build/arch/x86/include/generated/uapi', '-I/lib/modules/4.14.65+/build/include/uapi', '-I/lib/modules/4.14.65+/build/include/generated', '-I/lib/modules/4.14.65+/build/include/generated/uapi', '-I./arch/x86/include', '-Iarch/x86/include/generated/uapi', '-Iarch/x86/include/generated', '-Iinclude', '-I./arch/x86/include/uapi', '-Iarch/x86/include/generated/uapi', '-I./include/uapi', '-Iinclude/generated/uapi', '-include', './include/linux/kconfig.h', '-D__KERNEL__', '-D__HAVE_BUILTIN_BSWAP16__', '-D__HAVE_BUILTIN_BSWAP32__', '-D__HAVE_BUILTIN_BSWAP64__', '-Wno-unused-value', '-Wno-pointer-sign', '-fno-stack-protector'],
  'unsaved_files' : [('definitions.h', '...', 22), ('/bpftrace/include/__stddef_max_align_t.h', '...', 1771), ('/bpftrace/include/float.h', '...', 5192), ('/bpftrace/include/limits.h', '...', 3735), ('/bpftrace/include/stdarg.h', '...', 2025), ('/bpftrace/include/stddef.h', '...', 4499), ('/bpftrace/include/stdint.h', '...', 23388)],
  'options' : 0,
}
Clang error while parsing C definitions: 2
Input (22): #include <net/sock.h>

Unknown struct/union: 'sock'
exit status 1

Which is consistent with the error and failure mode of bpftrace/bpftrace#266.

To fix this, we should either use an ubuntu-based bpftrace that is dynamically linked, or if bpftrace figures out to build a statically linked bpftrace with ubuntu (currently ubuntu doesn't patch a static libclang.a), then use that instead. Of course, maybe someone will figure out a way to make the alpine build work, but I think the issue is in musl so I'm not particularly hopeful.

Until this is fixed, some scripts will randomly crash which makes this tool pretty unreliable.

Ephemeral containers seem a valid alternative for kubectl trace.

This issue is open so that we can explore them within kubectl trace and post the results

https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/

kubectl trace is operating against kubernetes APIs to run bpftrace programs.
In order to verify the correctness of such integration and provide a stable experience to our users we need to write integration tests.

Tasks to consider this issue closed (can be done in multiple PRs):

• Implement an integration testing framework (done in #26)
• Integrate with travis (done in #26)
• Integration tests for run (done in #26)
• Integration tests for run with attach
• Integration tests for get
• Integration tests for attach
• Integration tests for logs

Contributors are very welcome but while talking with people willing to contribute or contributing already it seems that getting started is difficult.

This task can be a good starting point for a new contributor willing to understand how the project works.

I think that using pods directly instead of jobs would reduce the overall complexity of this project and would make resource management easier.
Still exploring this idea, not 100% sure we want to replace them yet.

On my machine this is the mount output in the pod related to the trace.

/ # mount
...
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
debugfs on /sys/kernel/debug type debugfs (rw,relatime) <=
...
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime) <=
/ # command terminated with exit code 137

Anyway in other environments - eg., Docker for Mac (not my machine :P) - the sys directory lacks some subdirectories (for example those indicated here), thus it does not work.

Somehow related to #7.

I updated to go 1.13 and tried to install from master

16:04 $ go get -u github.com/iovisor/kubectl-trace/cmd/kubectl-trace
go: finding github.com/iovisor/kubectl-trace latest
go: github.com/iovisor/kubectl-trace/cmd/kubectl-trace imports
	github.com/iovisor/kubectl-trace/pkg/cmd imports
	github.com/iovisor/kubectl-trace/pkg/attacher imports
	k8s.io/kubernetes/pkg/kubectl/util/term imports
	github.com/docker/docker/pkg/term imports
	github.com/docker/docker/pkg/term/windows imports
	github.com/Sirupsen/logrus: github.com/Sirupsen/[email protected]: parsing go.mod:
	module declares its path as: github.com/sirupsen/logrus
	        but was required as: github.com/Sirupsen/logrus

Here is go info

16:16 $ go version
go version go1.13 darwin/amd64

I think it will be very useful if we could release pre-built binaries.

Symptoms:

• the job is created successfully
• its status stays empty (status: {})
• no pods get created
• I can see the following event with kubectl get events -n default:

3s Warning FailedCreate Job Error creating: pods "kubectl-trace-603733c8-17e6-11e9-aec8-c85b763781a4-" is forbidden: unable to validate against any pod security policy: [spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[2]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

kubectl-trace should use a specific ServiceAccount and its documentation should say how to install the ServiceAccount and its associated PSP.

Right now traces have a TTL that is set to 100 seconds,
https://github.com/fntlnz/kubectl-trace/blob/70140b3cbfd4dd2896ba1b1afed4f7caad28332a/pkg/tracejob/job.go#L208-L211

For most users this is ok but this should definitely be configurable with a flag to allow increasing or decreasing it.

I think the default should be run forever or until the trace exits.

I really like that kubectl-trace works as a standalone executable with no dependencies, and I think that's a desirable feature to keep.

Given the popularity of operators in the kubernetes community (and in our own infrastructure), I think that there is probably a good use-case for a different mode of operation for kubectl-trace that is custom-resource based.

For example, the current TraceJob struct is a nice blueprint of what a CRD for such an operator could be. The operator itself could just vendor in kubectl-trace code, and avoid duplication.

For a hackdays project, we built something very similar. One of the big draws with this is that kubect-trace doesn't have to be the only client for this - using a CR as the interface would allow for other tooling to plug into it very easily.

bpftrace just got support for positional parameters https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#9-1--n-positional-parameters

This means:

• the current hack we have to pass to the bpftrace program the process id using the $container_pid variable can be avoided and converted to that
• we need to check if this new way to handle parameters breaks anything before upgrading bpftrace to a version after this commit bpftrace/bpftrace@6da957a
• define a set of parameters we want to pass by default, like pod id, namespace etc..

At FOSDEM @alban showed the usage of kubectl trace with cgroups v2 in order to filter tasks in a container.

This issue is here to formalize that content in the readme so that everyone can take start using it for that use case.

There's some more context here: bpftrace/bpftrace#329 (comment)

All the kubectl commands can be used with the -v flag to specify verbosity, we need to have the same behavior and allow the user to increase the verbosity of klog using that flag.

As a user I expect the --all flag to work together with --all-namespaces, at the current moment this is not true, when I do:

kubectl trace get --all-namespaces
NAMESPACE       NODE            NAME                                                    STATUS          AGE
caturday        127.0.0.1       kubectl-trace-e0d048c8-2003-11e9-982a-8c164500a77e      <missing>       <missing>
caturday        127.0.0.1       kubectl-trace-e5b13530-2003-11e9-8db9-8c164500a77e      <missing>       <missing>

I expect to be able to delete them all with

kubectl trace delete --all --all-namespaces

But that commant doesn't select them all and returns:

an empty namespace may not be set when a resource name is provided

What I would expect is the traces to be all deleted like how it happens when I do that action specifying the namespace:

kubectl trace delete --all -n caturday     
trace job kubectl-trace-e0d048c8-2003-11e9-982a-8c164500a77e deleted
trace job kubectl-trace-e5b13530-2003-11e9-8db9-8c164500a77e deleted
trace configuration kubectl-trace-e0d048c8-2003-11e9-982a-8c164500a77e deleted
trace configuration kubectl-trace-e5b13530-2003-11e9-8db9-8c164500a77e deleted

The job container(s) should specify resource requests, otherwise environments that enforce this will not be able to schedule jobs:

  Type     Reason        Age    From            Message
  ----     ------        ----   ----            -------
  Warning  FailedCreate  3m37s  job-controller  Error creating: pods "kubectl-trace-d9c10bfa-2273-11e9-889a-000c2907600a-pktmb" is forbidden: failed quota: custom-resource-quotas: must specify limits.cpu,limits.memory,requests.memory
  Warning  FailedCreate  3m27s  job-controller  Error creating: pods "kubectl-trace-d9c10bfa-2273-11e9-889a-000c2907600a-hf4s9" is forbidden: failed quota: custom-resource-quotas: must specify limits.cpu,limits.memory,requests.memory
  Warning  FailedCreate  3m7s   job-controller  Error creating: pods "kubectl-trace-d9c10bfa-2273-11e9-889a-000c2907600a-v8qrp" is forbidden: failed quota: custom-resource-quotas: must specify limits.cpu,limits.memory,requests.memory
  Warning  FailedCreate  2m27s  job-controller  Error creating: pods "kubectl-trace-d9c10bfa-2273-11e9-889a-000c2907600a-wq8s4" is forbidden: failed quota: custom-resource-quotas: must specify limits.cpu,limits.memory,requests.memory
  Warning  FailedCreate  67s    job-controller  Error creating: pods "kubectl-trace-d9c10bfa-2273-11e9-889a-000c2907600a-52grj" is forbidden: failed quota: custom-resource-quotas: must specify limits.cpu,limits.memory,requests.memory

The fix here would be to set a relatively low request, and reasonable limits for cpu and memory so that the job can be reliably scheduled as a burstable pod.

I'm exploring the idea to allow users to add scheduled traces to this project.

A scheduled trace is a trace that runs at a specific time in a similar way a Cronjob works in kubernetes.

It will leave the traces it creates in the system so that the user can inspect them using kubectl trace logs (#4).

Since most of bpftrace commands need the user to stop the profiling by sending a signal this will also need to support a way to have a configurable timeout so that the signal is automatically sent.

The user will be able to send the signal by attaching to the current job.

This is my first day using kubectl-trace, so I apologize if this is a silly question/if my assessment of it is wrong.

output of kubectl trace version:
git commit: d34d1d5
build date: 2019-09-19 09:00:13 -0600 MDT

output of uname -r on the node:
5.4.0-1009-aws (Ubuntu 20.04 AMI)

Platform:
AWS (with kops)

Problem:

I've been able to successfully run this command:
kubectl trace run -e "tracepoint:syscalls:sys_enter_* { if(args->ret < 0) {@[ustack] = count();} }" pod/some-pod -a

However, when I try to run this command in a different namespace (on a pod in the kube-system namespace for example), I get:

if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
definitions.h:55:3: error: unknown type name 'key_serial_t'
definitions.h:123:3: error: unknown type name 'cap_user_header_t'
definitions.h:124:3: error: unknown type name 'cap_user_data_t'
definitions.h:133:3: error: unknown type name 'cap_user_header_t'
definitions.h:134:9: error: unknown type name 'cap_user_data_t'
definitions.h:375:9: error: unknown type name 'sigset_t'
definitions.h:1102:3: error: unknown type name 'aio_context_t'
definitions.h:1113:3: error: unknown type name 'aio_context_t'
definitions.h:1122:3: error: unknown type name 'aio_context_t'
definitions.h:1135:3: error: unknown type name 'aio_context_t'
definitions.h:1150:3: error: unknown type name 'aio_context_t'
definitions.h:1159:3: error: unknown type name 'aio_context_t'
definitions.h:1174:9: error: unknown type name 'sigset_t'
definitions.h:1987:3: error: unknown type name 'siginfo_t'
definitions.h:2071:9: error: unknown type name 'sigset_t'
definitions.h:2124:3: error: unknown type name 'rwf_t'
definitions.h:2229:3: error: unknown type name 'rwf_t'
definitions.h:2240:3: error: unknown type name 'qid_t'
definitions.h:2417:3: error: unknown type name 'key_serial_t'
fatal error: too many errors emitted, stopping now [-ferror-limit=]
exit status 1                                                           

Steps taken:

At first I thought it had something to do with trying to use the default service account, but following the instructions in the readme does not appear to make a difference.

Afterwards, I noticed this issue on bpftrace which seems to indicate that this is an issue with headers not being installed; however, running the same command with --fetch-headers appended yields:

WARNING: Cannot find distro-specific headers for "Ubuntu". Fetching generic headers.
++ uname -r
+ BUILD_DIR=/linux-generic-5.4.0-1009-aws
++ uname -r
+ SOURCES_DIR=/usr/src/linux-generic-5.4.0-1009-aws
+ '[' '!' -e /usr/src/linux-generic-5.4.0-1009-aws/.installed ']'
+ echo 'Installing kernel headers for generic kernel'
+ fetch_generic_linux_sources
Installing kernel headers for generic kernel
++ uname -r
+ kernel_version=5.4.0-1009-aws
++ echo 5.4.0-1009-aws
Fetching upstream kernel sources for 5.4.0-1009-aws.
+ major_version=5
+ echo 'Fetching upstream kernel sources for 5.4.0-1009-aws.'
+ mkdir -p /linux-generic-5.4.0-1009-aws
+ curl -sL https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.4.0-1009-aws.tar.gz
+ tar --strip-components=1 -xzf - -C /linux-generic-5.4.0-1009-aws
tar: invalid magic
tar: short read
real    0m0.271s
user    0m0.038s
sys    0m0.004s
+ generate_headers
Generating kernel headers
+ echo 'Generating kernel headers'
+ cd /linux-generic-5.4.0-1009-aws
+ zcat /proc/config.gz
zcat: /proc/config.gz: No such file or directory
+ make ARCH=x86 oldconfig
make: *** No rule to make target 'oldconfig'.  Stop.
+ make ARCH=x86 prepare
make: *** No rule to make target 'prepare'.  Stop.
+ find /linux-generic-5.4.0-1009-aws -regex '.*\.c\|.*\.txt\|.*Makefile\|.*Build\|.*Kconfig' -type f -delete
real    0m0.006s
user    0m0.005s
sys    0m0.000s
+ mv /linux-generic-5.4.0-1009-aws /usr/src
real    0m0.001s
user    0m0.001s
sys    0m0.000s
+ touch /usr/src/linux-generic-5.4.0-1009-aws/.installed
+ HEADERS_TARGET=/usr/src/linux-generic-5.4.0-1009-aws
++ uname -r
+ mkdir -p /lib/modules/5.4.0-1009-aws
++ uname -r
+ ln -sf /usr/src/linux-generic-5.4.0-1009-aws /lib/modules/5.4.0-1009-aws/source
++ uname -r
+ ln -sf /usr/src/linux-generic-5.4.0-1009-aws /lib/modules/5.4.0-1009-aws/build
+ touch /lib/modules/.installed
stream closed
kubectl-trace-67b1691a-9ae6-11ea-be33-acde48001122 if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
kubectl-trace-67b1691a-9ae6-11ea-be33-acde48001122 fatal error: '/lib/modules/5.4.0-1009-aws/source/include/linux/kconfig.h' file not found

When changing the bpftrace expression to just: "tracepoint:syscalls:sys_enter_* { @[ustack] = count(); }, then it works regardless of pod/namespace, which leads me to believe that it has something to do with the args parameter.

Apologies again if this the wrong place to ask this kind of question or if my assessment of my problem is incorrect.

We should add PrintFlags and use them to accordingly implement the printing as kubectl commands do.

We use kind for integration tests against a kubernetes cluster. We now have a dirty hack to load the built images into the kind docker daemon since when we started using it kind was at a very early stage and didn't have the load command. Now the load command is available so we can just replace the hack with it!

Docs here
https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster

At the moment of writing this issue, kubectl trace only supports X86-64 as target architecture.

The tool should be able to schedule bpftrace programs against all the architectures supported by bpftrace, that are X86-64 and arm64 (aarch64) see the Cmake definition.

To achieve that we need to build the trace runner image as a multi arch image.

Current -f flag should be implemented with genericclioptions.FileNameFlags (source) to make it behave as the same flag does for kubectl apply.

I tried to run a trace on a cluster built with kops, here is what I see in trace logs

# kubectl-trace logs kubectl-trace-c2405ead-a4b3-11e9-890b-0232f65d1adc
if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
/bpftrace/include/asm_goto_workaround.h:14:10: fatal error: 'linux/types.h' file not found

Relevant info

# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.9", GitCommit:"16236ce91790d4c75b79f6ce96841db1c843e7d2", GitTreeState:"clean", BuildDate:"2019-03-25T06:40:24
Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.9", GitCommit:"16236ce91790d4c75b79f6ce96841db1c843e7d2", GitTreeState:"clean", BuildDate:"2019-03-25T06:30:48
Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
# kubectl-trace version
git commit:
build date: undefined
# uname -a
Linux ip-10-65-134-126 4.9.0-7-amd64 #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x86_64 GNU/Linux

At the current moment bpftrace commands can be attached only against nodes in a kubernetes cluster as they are listed by kubectl get nodes.

PR #1 already introduces support for this in the cli system, however we still need to figure out how to implement that.

A way could be to use a shared process namespace ¹ between the user's pod and a container that is injected in the user's pod template instead of using a job as we do for nodes.

Tagging some people I discussed about this so that you can track this out guys: @calavera @leodido @gianarb

This is (I think) fairly easy to reproduce. If you specify the -a flag to kubectl trace run, it will often not attach to the trace, and you must manually execute kubectl trace attach.

For my case, It actually rarely happens that attach actually does work.

I wonder if it just needs to poll / retry more often or something, or if it can get a better signal about when it should be able to successfully attach.

When we use the log command on a completed job (pod in succeeded phase), the execution output is not displayed.

It may be nice to have the output also in this case.

Steps to reproduce

Run kubectl trace on a node

$ kubectl trace run kind-worker1 -e 'kprobe:do_sys_open { printf("%s: %s\n", comm, str(arg1)) }'
trace 43db844d-3ae6-11e9-8802-9cb6d0de1923 created

Trying to see the log on a running trace

$ kubectl trace log 43db844d-3ae6-11e9-8802-9cb6d0de1923
kubelet: /proc/sys/kernel/pid_max
kubelet: /sys/fs/cgroup/blkio/kubepods/blkio.io_serviced_recursive
kubelet: /sys/fs/cgroup/blkio/kubepods/blkio.throttle.io_service_bytes
kubelet: /sys/fs/cgroup/blkio/kubepods/blkio.throttle.io_serviced
kubelet: /sys/fs/cgroup/memory/kubepods/memory.stat
redis-server: /proc/1/stat

So far so good, it works fine.

Let's stop the trace now.

$ kubectl trace attach 43db844d-3ae6-11e9-8802-9cb6d0de1923
^C
first SIGINT received, now if your program had maps and did not free them it should print them out

Let's try to view the output again

$ kubectl trace log 43db844d-3ae6-11e9-8802-9cb6d0de1923

No output is displayed. 😒