awslabs / eks-node-viewer Goto Github PK

Right now only the cpu and memory requests are shown. It would be great if the real cpu and memory usage would also be shown.

Hello, can you please add tag for some initial release. I want to package eks-node-viewer for nixpkgs: NixOS/nixpkgs#210510. Thanks.

The viewer only displays public pricing.

Can a feature be added to see the estimated cost of nodes using Enterprise Discount Pricing?

Example output:

ip-10-0-123-34.eu-west-2.compute.internal  cpu █████████░░░░░░░░░░░░░░░░░░░░░░░░░░  25% (14 pods) r6i.xlarge - - Ready
•
←/→ page • q: quit
2023/11/09 20:00:33 updating spot pricing, ProcessProviderParseError: parse failed of credential_process output: {
    "Version": 1,
    "AccessKeyId": "[... YES IT IS PRINTING MY KEY!!! ...]",
    "SecretAccessKey": "[... YES IT IS PRINTING MY SECRET!!! ...]",
    "SessionToken": "[... YES IT IS PRINTING MY TOKEN!!! ...]",
    "Expiration": "2023-11-11T07:58:44+0100"
}

caused by: parsing time "2023-11-11T07:58:44+0100" as "2006-01-02T15:04:05Z07:00": cannot parse "+0100" as "Z07:00", using existing pricing data from 2022-12-05T13:09:08Z
2023/11/09 20:00:34 updating fargate pricing, ProcessProviderParseError: parse failed of credential_process output: {
    "Version": 1,
    "AccessKeyId": "[... YES IT IS PRINTING MY KEY!!! ...]",
    "SecretAccessKey": "[... YES IT IS PRINTING MY SECRET!!! ...]",
    "SessionToken": "[... YES IT IS PRINTING MY TOKEN!!! ...]",
    "Expiration": "2023-11-11T07:58:44+0100"
}

My laptop is currently running in CET timezone.

This is a great piece of software, thanks!

Could it be better if the visual aid's colors were reversed, meaning when the node is at 100% capacity, its red on the right vs on the left (which would be green). So at a glance, a user would know that if its red, the node is fully utilized.

Monthly pricing doesn't seem to match total hourly pricing * 365*24/12 🤔

Example:

$0.008/hour | $6.132/month

Where it should be: $5.84/month

Hi,

It seems to me that it could be useful to have an option to show costs and exit.

This way it could be used to quickly evaluate costs of Kubernetes cluster. Similar to infracost for Terraform. They have issue to add support for k8s, but it still in backlog, while eks-node-viewer already have this data.

Just jotting down a request, ideally this is distributed as a brew install.

Currently we can view the dynamic utilisation of node. But if we get the capabilities to view dynamic pod utilisation thats even more super useful

Hello,

Initially I was getting pricing info for both spot and on-demand capacity for Karpenter provisioned nodes. Was not getting capacity or pricing for other nodes in my cluster that were not provisioned by Karpenter. Now, I'm only getting pricing info for on-demand capacity nodes provisioned by Karpenter. Does eks-node-viewer not show pricing and capacity for non-karpenter provisioned nodes? Any idea why I'm not getting pricing for spot capacity Karpenter provisioned nodes?

eks-node-viewer version 0.4.3
commit: 23ccb3a
built at: 2023-08-07T15:16:40Z
built by: goreleaser

far too much precision of seconds

If a given session is expired or the iam auth is experiend eks-node-viewer should just quit. Right now it is brute force the api with messages like this:

	status code: 400, request id: 409ff304-8721-4c5d-9ab8-d2c2c10a5ca3; ExpiredTokenException: The security token included in the request is expired

It would be nice if I could see the memory in a different color. It throws me off when it's matching the cpu color.

Even better yet, the ability to choose a custom color would be pretty cool.

Right now the fargate stats are not correct because the provisioned instance is always bigger than the requested resources for the pod.

fargate-ip-10-8-137-244.eu-central-1.compute.internal cpu ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 10% (1 pods) 0.25vCPU-1GB/$0.0118 Fargate - Ready
memory ███████████████░░░░░░░░░░░░░░░░░░░░ 42%

│ Non-terminated Pods: (1 in total) │
│ Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age │
│ --------- ---- ------------ ---------- --------------- ------------- --- │
│ karpenter karpenter-5fdbd5d978-6jqcn 200m (10%) 200m (10%) 768Mi (42%) 768Mi (42%) 31d

The cpu usage should be something like 80%, the memory usage should be 78%.

You can only monitory cpu and memory (maybe other) resources.requests.

Can you add a feature to monitor the resources.limits as well?

Pricing not showing for m7i instance types both m7i and m7i-flex. This applies to on-demand and spot.

i had ENV running in a shell, and spun up 20 new pods/nodes and the CLI app updated to include the new nodes but not the stats about them

after I restarted the app all the expected details where there

Thanks for this tool, it's proving really useful for watching karpenter do its thing.

Would it be possible to include the node age as well as a data point? Would be handy for getting a realtime feel for how often spot instances are being reissued in the pool.

I installed using go install github.com/awslabs/eks-node-viewer/cmd/eks-node-viewer@latest

$ eks-node-viewer                       
2022/12/09 11:43:21 creating client, exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"

Credentials are correctly configured

Starting from version 0.4.0 eks-node-viewer crashes on start:

$ ~/bin/eks-node-viewer --version
eks-node-viewer version 0.4.2
commit: b1dcd24fdfbfcfb7e515d9366c876f644634f31f
built at: 2023-07-20T18:27:47Z
built by: goreleaser

$ ~/bin/eks-node-viewer
Caught panic:

runtime error: slice bounds out of range [:-6]

Restoring terminal...

goroutine 1 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.20.6/x64/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	/opt/hostedtoolcache/go/1.20.6/x64/src/runtime/debug/stack.go:16 +0x19
github.com/charmbracelet/bubbletea.(*Program).Run.func1()
	/home/runner/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:440 +0x95
panic({0x1d12cc0, 0xc000830090})
	/opt/hostedtoolcache/go/1.20.6/x64/src/runtime/panic.go:890 +0x263
github.com/awslabs/eks-node-viewer/pkg/model.(*UIModel).View(0xc0005b2240)
	/home/runner/work/eks-node-viewer/eks-node-viewer/pkg/model/uimodel.go:106 +0x7b4
github.com/charmbracelet/bubbletea.(*Program).Run(0xc000412240)
	/home/runner/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:487 +0x6f5
github.com/charmbracelet/bubbletea.(*Program).Start(...)
	/home/runner/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:548
main.main()
	/home/runner/work/eks-node-viewer/eks-node-viewer/cmd/eks-node-viewer/main.go:103 +0x94f

Version 0.3.0 working without issues

I have multiple configurations in my KUBECONFIG environment variable, diveded by a : as described in https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#append-home-kube-config-to-your-kubeconfig-environment-variable

Unfortunately, this breaks the eks-node-viewer with the following error:

$ eks-node-viewer
2022/12/14 09:08:56 creating client, stat .kubeconfig:/Users/[...]/.kube/config:/Users/[...]/.kube/config.d/01_preferences:/Users/[...]/.kube/config.d/EKS:/Users/[...]/.kube/config.d/local:[...]: no such file or directory

it would be great if the fargate container would also be shown with price info

I'm overwriding my kubeconfig path with env var KUBECONFIG but this one is no't used when use eks-node-viewer.

So i have to perform this : cat ~/.kube/k0s.cfg > ~/.kube/config everytime i use eks-node-viewer on a different context (as my context is defined in another file path than the default one)

KUBECONFIG=/home/xxxxxx/.kube/k0s.cfg

https://github.com/awslabs/eks-node-viewer#computed-labels

the docs means that to see those infos i have to label to nodes?
if then, what should be the value of them?

eks-node-viewer does an amazing job at displaying the resource allocation across nodes. The total no of pods shown on the right of cpu percentage includes the pods in non-running state which doesn't reflect the current resource allocation. Not sure if this is intended.

The node shown above shows incorrect resources allocation.

Cloud9 has go1.18.x, It is not installed. 1.20.x is same.

I saw the request for brew, just putting my two cent in for a krew distribution

Hello,

First of all , thanks to share this tool with community :-)

This is not an issue it's only a question, because I tested with a EKS cluster running inside AWS and the results don't match with others tool like top, htop etc....

As you can see in the screenshot I understand that the pods are using a lot of CPU but in other hand I can see that we are not using a lot of memory.

In the next screenshot we can see the result of HTOP command.

Here the result it's totally true compared with eks-node-viewer info, the memory is very low comparing with the total. --> 6.63GB of 30.9GB

But as you can see the CPU is not very high... for me is strange because in eks-node-viewer it said that is around 94%...

Here other screenshot about kubectl top nodes.

Maybe eks-node-viewer check the CPU sum for deployments?¿ Only take the limits? Because is reserved to the pods?¿

Please would be possible to explain a little bit how to interpreter the results?

Thanks!! :-)

When listing nodes in a cluster with lots of nodes it would be useful to make the list scrollable. Right now the only options are to zoom out until all nodes fit on a single page which can make text unreadable.

It'd be great to expose data as Prometheus metrics. So that we can have better visualization and/or alerts

It appears that eks-node-viewer currently sorts by node creation time, with oldest on top. Of all the options to sort by, creation time is probably the least interesting to me.

It would be very helpful to be able to choose other columns to sort by, and whether they are asc/desc, including arbitrary columns like the ones generated by --extra-labels. I label my nodes so I can see at a glance what provisioner they belong to, and being able to sort on the label column would be a huge help.

go install github.com/awslabs/eks-node-viewer/cmd/eks-node-viewer@latest
go: github.com/awslabs/eks-node-viewer/cmd/eks-node-viewer@latest (in github.com/awslabs/[email protected]): go.mod:5: unknown directive: toolchain

Hello,

Very nice tool, thank you for open-sourcing it!

It seems to me that it's now tries to fetch pricing information with default profile credentials and if it's not defined, throws errors like:

2022/12/12 10:49:49 updating on-demand pricing, NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors; NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors, using existing pricing data from 2022-10-31T16:47:25Z

I think that it would be useful to have command line option to define custom profile in session init.

using this tool when i migrate from the CAS to the Karpenter, sometimes i've watched that cpus or memories were available but more nodes created.

2 main reasons:

1 . the pod, would be scheduled, should be bounded in the specific AZ(since the PV).
2. the node had pods with the max # of pods of that.

imo, i have no idea to describe better for the first case. but the second case, the num of pods, is good to describe as progression bars, like other resource types.

both they are able to be checked with the label topology.kubernetes.io/zone and the computed label eks-node-viewer/node-pods-usage, but visual things would be better.

btw this tool is awesome. i really love this. thank you.

Hello, first of all thanks for the tool, it looks great.

The problem I'm facing is that even when I followed the instructions from the README to install the eks-node-viewer cli, after the installation the command is not available.

❯ go version
go version go1.19.4 darwin/amd64

I had to clone the repo and build it locally to make it work:

It appears that on spot instances the price estimate is using the on-demand price:

ip-10-63-242-25.ec2.internal  cpu    █████████████████████████████████░░  94% (14 pods) c6a.large/$0.076    On-Demand - Ready 
                              memory ███████████████████░░░░░░░░░░░░░░░░  54%                                                 
ip-10-63-247-246.ec2.internal cpu    ████████████████████████████░░░░░░░  79% (13 pods) c6a.large/$0.076    On-Demand - Ready 
                              memory ███████████████████░░░░░░░░░░░░░░░░  54%                                                 
ip-10-63-240-13.ec2.internal  cpu    ███████████████████████████████████  99% (18 pods) c6a.large           -         - Ready 
                              memory ████████████████████████████████░░░  91%                                                 
ip-10-63-243-187.ec2.internal cpu    ███████████████████████████████████ 100% (15 pods) c6a.large           -         - Ready 
                              memory ████████████████████████░░░░░░░░░░░  68%                                                 
ip-10-63-243-63.ec2.internal  cpu    ██████████████████████████████████░  97% (15 pods) c6in.large/$0.113   Spot      - Ready 
                              memory ███████████████████████████████░░░░  89%                                                 
ip-10-63-242-35.ec2.internal  cpu    █████████████████████████████████░░  94% (13 pods) c6i.large/$0.085    Spot      - Ready 
                              memory ██████████████████████░░░░░░░░░░░░░  64%                                                 
ip-10-63-245-172.ec2.internal cpu    ███████████████████████████████████  99% (21 pods) c6in.xlarge/$0.227  Spot      - Ready 
                              memory ███████████████████████████████████  99%                                                 
ip-10-63-240-143.ec2.internal cpu    ██████████████████████████████████░  97% (14 pods) r6i.large/$0.126    Spot      - Ready 
                              memory █████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  15%                                                 
ip-10-63-244-109.ec2.internal cpu    ██████████████████████████████████░  96% (20 pods) c6in.xlarge/$0.227  Spot      - Ready 
                              memory ██████████████████████████████████░  98%                                                 
ip-10-63-246-161.ec2.internal cpu    ███████████████████████████████████  99% (29 pods) r6id.2xlarge/$0.605 Spot      - Ready 
                              memory █████████░░░░░░░░░░░░░░░░░░░░░░░░░░  25%                                                 
ip-10-63-246-95.ec2.internal  cpu    ███████████████████████████░░░░░░░░  76% (13 pods) c6id.large/$0.101   Spot      - Ready 
                              memory ███████████████████░░░░░░░░░░░░░░░░  54%                                                 

For example the r6id.2xlarge should be more like $0.1608 per Hour

This was with the latest version installed with Go.

Hi Team,

EKS node viewer showing only on-demand pricing , but it is not showing spot pricing. The spot instance created by karpenter and I can able to see label "karpenter.sh/capacity-type | spot" in spot node.

can this read a config file so we store regular parameters?
--extra-labels=karpenter.sh/provisioner-name,topology.kubernetes.io/zone -resources memory,cpu,pods
if there was a $HOME/.eks_node_viewer with the contents:

extra-labels=karpenter.sh/provisioner-name,topology.kubernetes.io/zone 
resources=memory,cpu,pods

For each node, can you also display the corresponding NodePool it was provisioned? (Or optionally MNG name)

And also, provide a command line option to filter and retrieve nodes from a specific NodePool

Reason: Customers might have lot of NodePools, in which case, they would like to understand per NodePool pricing and capacity utilization.

I'm not sure what caused this error but it's possible it's because I'm on unstable conference wifi. I wanted to open an issue just in case it's repeatable.

I'm running against fairly large clusters (150+ nodes).

W0626 08:39:03.980356  101661 reflector.go:533] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:231: failed to list *v1.Pod: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 1; INTERNAL_ERROR; received from peer
I0626 08:39:03.980442  101661 trace.go:219] Trace[437753376]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:231 (26-Jun-2023 08:37:57.011) (total time: 66968ms):
Trace[437753376]: ---"Objects listed" error:stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 1; INTERNAL_ERROR; received from peer 66968ms (08:39:03.980)
Trace[437753376]: [1m6.968683122s] [1m6.968683122s] END
E0626 08:39:03.980469  101661 reflector.go:148] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:231: Failed to watch *v1.Pod: failed to list *v1.Pod: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 1; INTERNAL_ERROR; received from peer

This error would appear below the node list. Nodes and pages would show up but pricing and utilization data was missing. This error would repeat and never load usage or pricing data.

While watching a node-group update of a managed EKS 1.21 node group I noticed that new instances come up as spot instances with the correct spot price and switch to on-demand after half a minute (when the capacityType label get's added) but keep showing the cached spot price instead of the correct on-demand price.

I'm not sure if the node should keep an attribute "last capacityType" and remove the cached price when lastCapacityType != currentCapacityType as that's a corner case.

The easiest solution might be to just not cache the price of a node at all and look it up for every update from the cached instance price lists. Or Update it, when updating a node?
https://github.com/awslabs/eks-node-viewer/blob/main/cmd/eks-node-viewer/main.go#L150

This was running version 164e1c9

When using eks-node-viewer with credentials provided from AWS SSO, I'm seeing:

2023/02/16 16:07:40 updating on-demand pricing, NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors; NoCredentialProviders: no valid providers in chain. Deprecated.

Setting AWS_PROFILE does not solve the issue.

Looking at issues in aws-sdk-go, this seems to be related to aws-sdk-go v1 not playing well with SSO. It might be fixed in v2?

Currently to use the utility we can only use the ~/aws/credentials or the aws config, but is it also possible to use the built-in IAM role in the machine ?

As shown in the screenshot below, pricing is not showing up. Also the bars are not refreshing. It seems like it's just stuck.

eks-node-viewer version 0.1.3
commit: 868b27101807da01031c47cc467d5d8a8a0d62b4
built at: 2023-02-25T00:02:55Z
built by: goreleaser

I'm following the Autoscaling section in the EKS Immersion Workshop
The command to install eks-node-viewer is:
go install github.com/awslabs/eks-node-viewer/cmd/eks-node-viewer@latest
That fails with:
../../go/pkg/mod/github.com/awslabs/[email protected]/cmd/eks-node-viewer/main.go:44:12: pattern ATTRIBUTION.md: no matching files found

I edited the command to install the v0.1.0 version and that worked:
go install github.com/awslabs/eks-node-viewer/cmd/[email protected]

I curl file (https://github.com/awslabs/eks-node-viewer/releases/download/v0.6.0/eks-node-viewer_Darwin_all) and run then got error malformed Mach-o file: ./eks-node-viewer

how can I solve this?

Would it be possible to add the uptime of the node in the last column ?
It would be very interesting :)

At least the root/data volumes should be included in the cost prediction.

In big clusters (100+ nodes), I find myself not able to see all the nodes even though I keep my terminal's font size to the bare minimum I need to be able to read. If I try to scroll the UI, eks-node-viewer gets terminated. It would be great to have a way to scroll through the UI.

To avoid people needing to build binaries and have the toolchaing installed locally, this repo should provide prebuild binary releases for major platforms via GH Releases.
i.e. https://goreleaser.com/