[bug] Global Network Utilization about grafana-dashboards-kubernetes HOT 9 CLOSED

Hi @reefland, sorry for the delay, will look at it this week!

from grafana-dashboards-kubernetes.

Hi @reefland,
Thanks for reporting this, I will check on my setup and come back to you.

from grafana-dashboards-kubernetes.

Here's a first fix: 6e3a73f

This should have been done when the labels got dropped from kube-prometheus-stack.

Let me know if this solves the issue.

from grafana-dashboards-kubernetes.

Global Network Utilization looks reasonable now, by Namespace is still bonkers:

from grafana-dashboards-kubernetes.

After re-working my install, this looks different now. I get the per namespace metrics now.

But still puzzled on why the per namespace numbers are way higher than the global numbers. My assumption is this is the difference between local and host networking? Here is the zoomed in version of by Namespaces:

The top 3 are ones that either use Host Networking or talk external to the cluster: ceph is just cluster to cluster member, but uses host networking so I can allow non-cluster computer to use the Rook-Ceph storage if needed. The CSI is using storage to a TrueNAS server and the Monitoring namespace is the only namespace still using that CSI (everything else is using Ceph).

I suspect there is some double counting going on.. the CSI is twice the PVC usage.. the containers virtual adapter and physical adapter? If I look at the underlying query rate(container_network_receive_bytes_total[2m0s]) with just the by namespace part removed, you see the results include multiple network adapters (virtual and physical) and I think this is where the double-counting comes in.

Trying to make the query simpler, I limit it to one namespace and one node of that namespace rate(container_network_receive_bytes_total{namespace="democratic-csi",instance="k3s01"}[2m0s]). You can see the multiple interfaces being counted:

If I make the global based on container_network_receive_bytes_total and container_network_transmit_bytes_total, at least it matches:

So maybe that is the way to do global? 🤷‍♂️

from grafana-dashboards-kubernetes.

Agree, something is probably counted more than once, and this is probably because of the current label selectors.
Sometimes, we need to narrow the scope, but in this case, I can't filter on the interfaces names because they are subject to change depending on the setup…

I made the benches a long time ago and it seemed correct, so I either missed something, or something changed since then.

I don't have the time to make the measures right now, but will have a look when I can.

from grafana-dashboards-kubernetes.

Blocked @jud336 for spamming issues and commits.

from grafana-dashboards-kubernetes.

Because the device names can be anything, It's not a good idea to filter on them, but I tried to exclude common virtual patterns to make the panels slightly more clear (we could build upon that).

Check out the latest version that sum by device, and let me know what you think.

Commit: 3e71721

from grafana-dashboards-kubernetes.

Closing

from grafana-dashboards-kubernetes.