cyberark / kubeletctl Goto Github PK

In version 1.18, the /metrics/resources/v1alpha1 endpoint will be going away.

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md

resource metrics endpoint /metrics/resource/v1alpha1 as well as all metrics under this endpoint have been deprecated. Please convert to the following metrics emitted by endpoint /metrics/resource: - scrape_error --> scrape_error - node_cpu_usage_seconds_total --> node_cpu_usage_seconds - node_memory_working_set_bytes --> node_memory_working_set_bytes - container_cpu_usage_seconds_total --> container_cpu_usage_seconds - container_memory_working_set_bytes --> container_memory_working_set_bytes - scrape_error --> scrape_error (#86282, @RainbowMango) [SIG Node]

Summary

I can successfully execute run command, but I failed to execute exec command with error unable to upgrade connection: Unauthorized

Steps to Reproduce

Steps to reproduce the behavior:

1. login to any node
2. try execute ls command in a container

./kubeletctl --cacert /etc/kubernetes/pki/ca.crt --cert /var/lib/kubelet/pki/kubelet-client-current.pem --key /var/lib/kubelet/pki/kubelet-client-current.pem -s 10.10.94.147  exec -p hello-world -c hello ls

Expected Results

It should print directory contents under /

Actual Results (including error logs, if applicable)

Failed with error unable to upgrade connection: Unauthorized

While I can successfully execute run command:

./kubeletctl --cacert /etc/kubernetes/pki/ca.crt --cert /var/lib/kubelet/pki/kubelet-client-current.pem --key /var/lib/kubelet/pki/kubelet-client-current.pem -s 10.10.94.147 run -p hello-world -c hello ls

Reproducible

• Always
• Sometimes
• Non-Reproducible

Version/Tag number

• kubeletctl: 1.7
• Kubernetes: v1.18.3
• Docker: 19.03.14

Environment setup

Our K8s nodes are running on VM in local machines.

K8s 1.25 introduced the Checkpoint API https://betterprogramming.pub/backup-and-restore-of-containers-with-kubernetes-checkpointing-api-e310cf29cd4a. It would be great to have support in kubeletctl

Adding a --token flag would improve usability, at least for me. The command would then look something like:

$ kubeletctl --token=<token> --server=<addr> <command>

Using a token is probably also possible if you create an kubeconfig file and then use the --config flag, but I'd rather not do that.

If the kubelet --client-ca-file option is used to require a client certificate, then kubeletctl needs a way to specify client cert and key files (analogous to curl --cert and --key arguments).

Summary

Due to a possible misconfiguration on my side of kubeletctl or perhaps another reason, the port being used to comunicate with the kubelet API is not correct. The port being used (39261) is the cluster port specified on my kubeconfig file (see Environment setup section).

Steps to Reproduce

Steps to reproduce the behavior:

1. Download the precompiled kubeletctl binary with:

$ curl -LO https://github.com/cyberark/kubeletctl/releases/download/v1.6/kubeletctl_linux_amd64 && chmod a+x ./kubeletctl_linux_amd64 && mv ./kubeletctl_linux_amd64 /usr/local/bin/kubeletctl

2. Try to check worker node kubelet's health:

$ kubeletctl -s 172.18.0.3 healthz

👀 NOTE:
The target worker node has the IP 172.18.0.3.

Expected Results

Get the proper output from the kubelet. In this case the endpoint tested was healthz. This is the output obtained using curl instead of kubeletctl:

$ curl -k https://172.18.0.2:10250/healthz
ok

Actual Results (including error logs, if applicable)

Using the default port for kubelet (port 10250) or setting it manually both result in the wrong address being used.

With the default port:

$ kubeletctl -s 172.18.0.3 healthz
[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch
[*] Failed to run HTTP request with error: Get "https://172.18.0.3:39261/healthz/": dial tcp 172.18.0.3:39261: connect: connection refused

$ kubeletctl -s 172.18.0.3 --port 10250 healthz
[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch
[*] The reponse failed with status: 404
[*] Message: 404 page not found

Reproducible

• Always
• Sometimes
• Non-Reproducible

Version/Tag number

Product version is the following:

$ kubeletctl version
[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch

 _           _           _                         _  
| |         | |         | |         _          _  | | 
| |  _ _   _| |__  _____| | _____ _| |_ ____ _| |_| | 
| |_/ ) | | |  _ \| ___ | || ___ (_   _) ___|_   _) | 
|  _ (| |_| | |_) ) ____| || ____| | |( (___  | |_| | 
|_| \_)____/|____/|_____)\_)_____)  \__)____)  \__)\_)

Author: Eviatar Gerzi
Version: 1.6

Environment setup

Running on local development box:

$ uname -a
Linux <box-name> 5.0.0-32-generic #34~18.04.2-Ubuntu SMP Thu Oct 10 10:36:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes version and cluster info:

$ kubectl cluster-info 
Kubernetes master is running at https://127.0.0.1:39261
KubeDNS is running at https://127.0.0.1:39261/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Config file pointed by $KUBECONFIG env var:

$ cat $KUBECONFIG
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: BASE64_CERTIFICATE_AUTHORITY_DATA
    server: https://127.0.0.1:39261
  name: kind-cluster-name
contexts:
- context:
    cluster: kind-cluster-name
    user: kind-cluster-name
  name: kind-cluster-name
current-context: kind-cluster-name
kind: Config
preferences: {}
users:
- name: kind-cluster-name
  user:
    client-certificate-data: BASE64_ENCODED_CERTIFICATE_DATA
    client-key-data: BASE64_ENCODED_KEY_DATA

Additional Information

The kubernetes flavor used for this scenario is KinD

The extra configuration for the worker nodes is the following:

- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        anonymous-auth: "true"
        authorization-mode: "AlwaysAllow"

This is mainly to allow unauthenticated requests to the kubelet api.

Kubelets that enforce authorization have known responses for permission errors. For example, if a request is authenticated but isn't authorized, the Kubelet will respond with: Forbidden (user=%s, verb=%s, resource=%s, subresource=%s)

kubeletctl currently prints a generic error message for permission errors, so it can be hard to understand what's causing the error:

$ kubeletctl --token=<token> pods
[*] Failed to run HTTP request with error: invalid character 'F' looking for beginning of value

The actual error can be seen using --raw, but if you're not using kubeletctl regularly it's easy to forget about that. It would be awesome if kubeletctl could check whether a response is a known permission error, and if so print the full error by default.

Kubelet responses for permission errors

• Unauthenticated request or Authentication error - http.StatusUnauthorized (401), Unauthorized
• Unauthorized request - http.StatusForbidden (403), Forbidden (user=%s, verb=%s, resource=%s, subresource=%s)
• Authorization error - http.StatusInternalServerError (500), Authorization error (user=%s, verb=%s, resource=%s, subresource=%s)

Hi
How can I make it work with my k8s on google cloud, or any other cloud

tls: failed to find any PEM data in certificate input

this is all I get back... my KUBECONFIG is properly set, I
daily work with it switching between several configurations

seems the kubeletctl is not handling this yaml section properly

clusters:
- cluster:
    certificate-authority-data: xyz

Add support for Linux ARM64

Summary

Provide brief overview and context for the discovered bug.

Steps to Reproduce

1. curl -LO https://github.com/cyberark/kubeletctl/releases/download/v1.11/kubeletctl_darwin_amd64 && chmod a+x ./kubeletctl_darwin_amd64 && mv ./kubeletctl_darwin_amd64 /usr/local/bin/kubeletctl
2. kubeletctl -s 10.11.12.13 metrics cadvisor
3. See error

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1763c4e]

goroutine 1 [running]:
kubeletctl/cmd.PrintPrettyHttpResponse(0x0, {0xc0000f17d0?, 0xc00021fd48?})
	/home/cyber/kubeletctl/cmd/print.go:56 +0x2e
kubeletctl/cmd/metrics.glob..func1(0x1f0d520?, {0x18c600e?, 0x4?, 0x4?})
	/home/cyber/kubeletctl/cmd/metrics/cadvisor.go:45 +0x4f
github.com/spf13/cobra.(*Command).execute(0x1f0d520, {0xc000324f00, 0x4, 0x4})
	/home/cyber/kubeletctl/vendor/github.com/spf13/cobra/command.go:842 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0x1f0cd40)
	/home/cyber/kubeletctl/vendor/github.com/spf13/cobra/command.go:943 +0x37d
github.com/spf13/cobra.(*Command).Execute(...)
	/home/cyber/kubeletctl/vendor/github.com/spf13/cobra/command.go:883
kubeletctl/cmd.Execute()
	/home/cyber/kubeletctl/cmd/root.go:83 +0x25
main.main()
	/home/cyber/kubeletctl/main.go:20 +0x17

Expected Results

It should work?

Actual Results

Throws panic.

Reproducible

• Always
• Sometimes
• Non-Reproducible

Version/Tag number

1.11

K8s: v1.23.7

Environment setup

macOS

Additional Information

Add any other context about the problem here.

kubeletctl commands that require arguments cause a panicif no arguments are provided. I confirmed this behaviour with this release.

$ ./kubeletctl --http --server REDACTED --port 10255 debug
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
kubeletctl/cmd/proxy/debug.glob..func1(0x15708a0, 0xc00034e320, 0x0, 0x5)
	/home/ubuntu/test/kubeletctl/cmd/proxy/debug/debug.go:89 +0x1e1
github.com/spf13/cobra.(*Command).execute(0x15708a0, 0xc00034e2d0, 0x5, 0x5, 0x15708a0, 0xc00034e2d0)
	/home/ubuntu/test/kubeletctl/vendor/github.com/spf13/cobra/command.go:842 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0x156dea0, 0x443cba, 0x151f1a0, 0xc000000180)
	/home/ubuntu/test/kubeletctl/vendor/github.com/spf13/cobra/command.go:943 +0x317
github.com/spf13/cobra.(*Command).Execute(...)
	/home/ubuntu/test/kubeletctl/vendor/github.com/spf13/cobra/command.go:883
kubeletctl/cmd.Execute()
	/home/ubuntu/test/kubeletctl/cmd/root.go:66 +0x31
main.main()
	/home/ubuntu/test/kubeletctl/main.go:20 +0x20

Panic comes from the following snippet:

var inputArgs string
if args == nil {
    fmt.Println("[*] No debug profile was specified")
    os.Exit(1)
} else {
    inputArgs = args[0]
}

The error comes from checking if the arguments slice is nil instead of checking the number of arguments (length of a nil slice is 0 too, see the playground). If no arguments are provided, cobra passes an empty slice as argument, not nil.

I confirmed this behaviour in the debug and exec commands, but it might be present in more of them.

I built this small repro so you can check cobra's behaviour when no arguments are provided:

package main

import (
	"fmt"

	"github.com/spf13/cobra"
)

func main() {
	cmdPrint := &cobra.Command{
		Use: "print [str]",
		Run: func(_ *cobra.Command, args []string) {
			fmt.Printf("args: %#+v\n", args)
		},
	}

	rootCmd := &cobra.Command{Use: "app"}
	rootCmd.AddCommand(cmdPrint)
	rootCmd.Execute()
}

:/tmp/example$ go run main.go print
args: []string{}
:/tmp/example$ go run main.go print foo
args: []string{"foo"}

Regards,
Miguel

The kubelet readonly server (normally at :10255) listens for HTTP. The kubelet main server (normally at :10250) will also listen for HTTP when no TLS configuration is specified.
When attempting to issue commands through kubeletctl to a kubelet's HTTP endpoint, the following error occurs:

$ kubeletctl --server 127.0.0.1 --port 10255 pods
[*] Failed to run HTTP request with error: Get https://127.0.0.1:10255/pods: http: server gave HTTP response to HTTPS client

$ curl http://127.0.0.1:10255/pods # works
{"kind":"PodList","apiVersion":"v1", ... }

Consider adding support for HTTP, perhaps through a cmdline flag.

Also, great tool (:

Summary

kubeletctl scan rce -s SERVER is not showing "+" on RCE column even though I can RCE

Steps to Reproduce

1. Setup microk8s cluster for testing
2. allow anonymous Kubelet API access
3. try to access https://SERVER:10250/pods to check for pods-json
4. if you are allowed to see pods-json try to scan kubeletctl scan rce -s SERVER
5. also check if you can execute commands

Expected Results

if I am allowed to kubeletctl exec "ls /" -c CONTAINER -p POD -s SERVER I should see a + in the "scan RCE" list

Actual Results

I have only "-" signs on kubeletctl scan rce -s SERVER but I can execute code

┌───────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                    Node with pods vulnerable to RCE                                   │
├───┬───────────┬─────────────────────────────────────────┬─────────────┬─────────────────────────┬─────┤
│   │ NODE IP   │ PODS                                    │ NAMESPACE   │ CONTAINERS              │ RCE │
├───┼───────────┼─────────────────────────────────────────┼─────────────┼─────────────────────────┼─────┤
│   │           │                                         │             │                         │ RUN │
├───┼───────────┼─────────────────────────────────────────┼─────────────┼─────────────────────────┼─────┤
│ 1 │ 10.0.2.15 │ website-k8s-675dd9956d-qj58f            │ default     │ website-k8s             │ -   │
└───┴───────────┴─────────────────────────────────────────┴─────────────┴─────────────────────────┴─────┘

shows "-" on RCE column, but if I try to execute code I can do:

┌──(root@kali)-[/home/kali]
└─# kubeletctl exec "ls /" -p website-k8s-675dd9956d-qj58f -c website-k8s -s 10.0.2.15
bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr

I also saw in source code that there is POST request to check manually and this is my output:

                               
┌──(root@kali)-[/home/kali]
└─# curl -k -XPOST https://10.0.2.15:10250/run/default/website-k8s-675dd9956d-qj58f/website-k8s -d "cmd=ls /"
rpc error: code = Unknown desc = failed to exec in container: failed to start exec "2766ae987637b8f679d7f68cbe02868c5dad0af36a08e8ed961825a274ac444d": OCI runtime exec failed: exec failed: unable to start container process: exec: "": executable file not found in $PATH: unknown       

Reproducible

• Always
• Sometimes
• Non-Reproducible

Version/Tag number

Version 1.11

Environment setup

• Running in self-hosted Linux (Debian) VirtualBox and installed MicroK8S
• Which cloud provider? Which container orchestrator (including version)?