Pod metrics not being collected using Bottlerocket about aws-otel-collector HOT 8 CLOSED

Hello @ElvenSpellmaker, can you share your complete deployment template. Also please refer to the v0.35.0 for the breaking change - #2317
Please try adding https://github.com/aws-observability/aws-otel-collector/blob/main/deployment-template/eks/otel-container-insights-infra.yaml#L189-L191 to run as a root user to collect pod metrics.

from aws-otel-collector.

Hey @vasireddy99, we're currently deploying the OTEL Helm chart here: https://github.com/aws-observability/aws-otel-helm-charts

Version: 0.21.0

What's interesting is that the chart says the containers run as root, but the issue you linked says the chart needs to run as aoc.

from aws-otel-collector.

Here are the Helm Values:

global:
  namespaceOverride: "aws-adot-otel"
awsRegion: "eu-west-2"
clusterName: "foo-bar-baz"
adotCollector:
  daemonSet:
    namespace: "aws-adot-otel"
    serviceAccount:
      name: "aws-adot-otel-sa"
      annotations:
        eks.amazonaws.com/role-arn: "arn:...:nonprod-aws-adot-otel-aws-adot-otel-sa"
    service:
      metrics:
        receivers: ["awscontainerinsightreceiver"]
        exporters: ["awsemf"]

The helm chart creates the K8s Service account with the permissions as stated in the helm chart. The Role in AWS eks.amazonaws.com/role-arn is created by us with the following permissions (through Terraform):

from aws-otel-collector.

I also notice the helm chart doesn't set any securityContext on the chart values: https://github.com/aws-observability/aws-otel-helm-charts/blob/main/charts/adot-exporter-for-eks-on-ec2/values.yaml

from aws-otel-collector.

I can confirm that manually adding the runAsUser and runAsGroup as 0 to the Dæmonset removes the error.
The Helm chart needs updating to allow us to specify a secuirty context.

However, I can't see any additional metrics since the error has gone away compared to when the error was present...

EDIT: The error message goes but it's still only collecting service metrics and not pod metrics, the default chart says it collects several pod metrics but it's just not (without any errors too)
EDIT2: I'm noticing this: no pod information is found in podstore for pod ..., so the root only fixes part of the issue.

EDIT3: Boom, if you add both the securityContext and the /run/dockershim.sock mount then pod metrics start to come in.

From: docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-troubleshooting.html#ContainerInsights-troubleshooting-bottlerocket

It seems the helm chart needs a few updates to allow this.

EDIT4: It seems that while it's collecting the metrics it's still not surfacing those in Container Insights for some reason...

from aws-otel-collector.

It seems that while it's collecting the metrics it's still not surfacing those in Container Insights for some reason...

I have same issue. After setting security context and fix /run/dockershim.sock for bottlerocket, some metrics remain unpublished.
is it because some metrics are available only with Container Insights with enhanced observability for Amazon EKS as listed in this docs?
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-EKS.html

from aws-otel-collector.

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

from aws-otel-collector.

This issue was closed because it has been marked as stale for 30 days with no activity.

from aws-otel-collector.