Versions

Charts:

- name: opentelemetry-collector
  version: 0.8.1
  repository: https://open-telemetry.github.io/opentelemetry-helm-charts
- name: refinery
  version: 1.3.0
  repository: https://honeycombio.github.io/helm-charts

Helm: ArgoCD (Helm 3 compatible)
Kubernetes cluster: 1.19 EKS

Steps to reproduce

1. Scale out a single Otel Collector to 2 or more replicas (behind and NLB) and point them at a K8s Refinery cluster in the same K8s cluster:

opentelemetry-collector:
  fullnameOverride: opentelemetry-collector
  image:
    repository: otel/opentelemetry-collector-contrib
    tag: 0.37.1
  command:
    name: otelcontribcol
    extraArgs: []
  agentCollector:
    enabled: false
  standaloneCollector:
    enabled: true
    replicaCount: 4
    resources:
      requests:
        cpu: 1
        memory: 1Gi
      limits:
        cpu: 1
        memory: 1Gi
  service:
    type: LoadBalancer
    annotations:
      external-dns.alpha.kubernetes.io/hostname: host.domain.name
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <some acm cert>
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "4317,4318,9411,14250,14268"
      service.beta.kubernetes.io/aws-load-balancer-alpn-policy: HTTP2Preferred
  ports:
    otlp:
      enabled: true
      containerPort: 4317
      servicePort: 4317
      hostPort: 4317
      protocol: TCP
    otlphttp:
      enabled: true
      containerPort: 4318
      servicePort: 4318
      hostPort: 4318
      protocol: TCP
    jaeger-compact:
      enabled: false
    jaeger-thrift:
      enabled: true
      containerPort: 14268
      servicePort: 14268
      hostPort: 14268
      protocol: TCP
    jaeger-grpc:
      enabled: true
      containerPort: 14250
      servicePort: 14250
      hostPort: 14250
      protocol: TCP
    zipkin:
      enabled: true
      containerPort: 9411
      servicePort: 9411
      hostPort: 9411
      protocol: TCP
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    exporters:
      otlp/refinery:
        endpoint: opentelemetry-collector-refinery:4317
        tls:
          insecure: true
        headers:
          x-honeycomb-team: <api key>
          x-honeycomb-dataset: <env>
      otlp/honeycomb-metrics:
        endpoint: api.honeycomb.io:443
        headers:
          x-honeycomb-team: <api key>
          x-honeycomb-dataset: <env>-metrics
    service:
      pipelines:
        traces:
          receivers:
            - otlp
          processors: 
            - memory_limiter
            - batch
          exporters:
            - otlp/refinery
        metrics:
          receivers:
            - otlp
            - prometheus
          processors: 
            - memory_limiter
            - batch
          exporters:
            - otlp/honeycomb-metrics
        logs:
          receivers:
            - otlp
          processors: 
            - memory_limiter
            - batch

2. Add a field to the Refinery rule field list

# original rules:
  rules:
    GoalSampleRate: 1
    Weight: 0.5
    FieldList:
      - http.method
      - http.status
      - service.name
      - messaging.kafka.partition
      - messaging.destination
      
# new rules:
  rules:
    GoalSampleRate: 1
    Weight: 0.5
    FieldList:
      - http.method
      - http.status
      - service.name
      - messaging.kafka.partition
      - messaging.destination
      - rpc.grpc.status_code

3. Observe context deadline exceeded messages in the Otel Collector logs like:

Exporting failed. Will retry the request after interval
error: failed to push trace data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded

4. Workaround is to manually delete Refinery pods one at a time. However, this results in a several minute time-to-recovery. May also need to delete Otel Collector pods if they continue to log errors.

Currently, the OTLP exporter used to send data into Honeycomb does not expose the insecure property. This is necessary if a downstream component (ie: refinery or secure tenancy) is not deployed with SSL on ingest.

Is your feature request related to a problem? Please describe.
If you have an existing way to manage secrets, the helm chart will overwrite your secrets.

Describe the solution you'd like
Offer optional "existingSecret" config property that will allow you to specify your secrets and not create them in the secrets.yaml.

Describe alternatives you've considered

Additional context

The opentelemetry-collector and this chart exposes Prometheus metrics on port 8888. Many charts add an option to generate a ServiceMonitor for users of the Prometheus Operator. We are using Honeycomb to observe our services but Prometheus for the infrastructure.

I think the kured chart has a nice api for generating the ServiceMonitor.

Versions

Chart: opentelemetry-collector-1.0.0
Helm: v3.7.1
Kubernetes cluster: 1.20.12-gke.1500

Steps to reproduce

1. Set the following in values.yaml

image:
  repository: otel/opentelemetry-collector
  tag: "0.41.0"

Additional context

helm get manifest otel-collector

---
# Source: opentelemetry-collector/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector
  labels:
    helm.sh/chart: opentelemetry-collector-1.0.0
    app.kubernetes.io/name: opentelemetry-collector
    app.kubernetes.io/instance: otel-collector
    app.kubernetes.io/version: "0.36.0"
    app.kubernetes.io/managed-by: Helm
---
# Source: opentelemetry-collector/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector
  labels:
    helm.sh/chart: opentelemetry-collector-1.0.0
    app.kubernetes.io/name: opentelemetry-collector
    app.kubernetes.io/instance: otel-collector
    app.kubernetes.io/version: "0.36.0"
    app.kubernetes.io/managed-by: Helm
data:
  opentelemetry-collector-config: |
    receivers:
      jaeger:
        protocols:
          grpc: null
          thrift_binary: null
          thrift_compact: null
          thrift_http: null
      otlp:
        protocols:
          http:
            cors:
              allowed_origins:
              - https://*.domain.com
              max_age: 7200
      zipkin: null

  
      batch: null
      memory_limiter:
        check_interval: 5s
        limit_mib: 1500
        limit_percentage: 75
        spike_limit_mib: 512
        spike_limit_percentage: 25

    exporters:
      otlp:
        endpoint: api.honeycomb.io:443
        headers:
          "x-honeycomb-team": "${HONEYCOMB_API_KEY}"
          "x-honeycomb-dataset": "dataset"

    extensions:
      health_check:
        port: 13133
      memory_ballast:
        size_in_percentage: 35
        size_mib: 683

    service:
      extensions:
      - health_check
      - memory_ballast
      pipelines:
        traces:
          exporters:
          - otlp
---
# Source: opentelemetry-collector/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  labels:
    helm.sh/chart: opentelemetry-collector-1.0.0
    app.kubernetes.io/name: opentelemetry-collector
    app.kubernetes.io/instance: otel-collector
    app.kubernetes.io/version: "0.36.0"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
    - name: jaeger-binary
      port: 6832
      targetPort: jaeger-binary
      protocol: UDP
    - name: jaeger-compact
      port: 6831
      targetPort: jaeger-compact
      protocol: UDP
    - name: jaeger-grpc
      port: 14250
      targetPort: jaeger-grpc
      protocol: TCP
    - name: jaeger-http
      port: 14268
      targetPort: jaeger-http
      protocol: TCP
    - name: otlp-grpc
      port: 4317
      targetPort: otlp-grpc
      protocol: TCP
    - name: otlp-http
      port: 4318
      targetPort: otlp-http
      protocol: TCP
    - name: zipkin
      port: 9411
      targetPort: zipkin
      protocol: TCP
  selector:
    app.kubernetes.io/name: opentelemetry-collector
    app.kubernetes.io/instance: otel-collector
---
# Source: opentelemetry-collector/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  labels:
    helm.sh/chart: opentelemetry-collector-1.0.0
    app.kubernetes.io/name: opentelemetry-collector
    app.kubernetes.io/instance: otel-collector
    app.kubernetes.io/version: "0.36.0"
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: opentelemetry-collector
      app.kubernetes.io/instance: otel-collector
  template:
    metadata:
      annotations:
        checksum/config: a91e3987fb1fdcc78aeab88c55c33a190e7e5f7a27a19b594064d8d147673151
      labels:
        app.kubernetes.io/name: opentelemetry-collector
        app.kubernetes.io/instance: otel-collector
    spec:
      serviceAccountName: otel-collector
      securityContext:
        {}
      containers:
        - name: opentelemetry-collector
          securityContext:
            {}
          image: "otel/opentelemetry-collector:0.41.0"
          imagePullPolicy: IfNotPresent
          command:
            - "/otelcontribcol"
            - "--config=/conf/opentelemetry-collector-config.yaml"
          env:
            - name: HONEYCOMB_API_KEY
              valueFrom:
                secretKeyRef:
                  name: honeycomb-api
                  key: api-key
          ports:
            - name: jaeger-binary
              containerPort: 6832
              protocol: UDP
            - name: jaeger-compact
              containerPort: 6831
              protocol: UDP
            - name: jaeger-grpc
              containerPort: 14250
              protocol: TCP
            - name: jaeger-http
              containerPort: 14268
              protocol: TCP
            - name: otlp-grpc
              containerPort: 4317
              protocol: TCP
            - name: otlp-http
              containerPort: 4318
              protocol: TCP
            - name: zipkin
              containerPort: 9411
              protocol: TCP
          volumeMounts:
            - name: opentelemetry-collector-config
              mountPath: /conf
          livenessProbe:
            httpGet:
              path: /
              port: 13133
          readinessProbe:
            httpGet:
              path: /
              port: 13133
          resources:
            limits:
              cpu: 1000m
              memory: 2Gi
            requests:
              cpu: 200m
              memory: 400Mi
      volumes:
        - name: opentelemetry-collector-config
          configMap:
            name: otel-collector
            items:
              - key: opentelemetry-collector-config
                path: opentelemetry-collector-config.yaml

The otel chart v1.0.0 is release with opentelemetry-collector-contrib:0.36.0 but I wanted to try opentelemetry-collector:0.41.0

But the containers fails to run because opentelemetry-collector-contrib docker container uses /otelcontribcol command and opentelemetry-collector docker container uses /otelcorecol.

Unfortunately the're no way with the current chart to set the command in deployment.yaml.

PS: If you confirm this bug I'd like to work on a fix

We place the configurations we override for all Helm charts we deploy into source control. Essentially we check in the name/version of the chart along with an overrides.yaml. Therefore we don't want to supply any secrets in values.yaml and instead use another process to deploy any needed secret resources.

Would you accept a PR to make apiKey an optional field? If the apiKey is not specified then the secret resource is not deployed and users of your chart would be required to deploy their own secret resource.

Something like:

{{- if and (ne .Values.honeycomb.apiKey "YOUR_API_KEY") }}
apiVersion: v1
kind: Secret
metadata:
  name: honeycomb-{{ include "opentelemetry-collector.fullname" . }}
  labels:
  {{- include "opentelemetry-collector.labels" . | nindent 4 }}
type: Opaque
data:
  api-key: {{ .Values.honeycomb.apiKey | b64enc | quote }}
{{- end }}

Is your feature request related to a problem? Please describe.

When setting up ingresses for a Helm deployment of Refinery with both a gRPC and HTTP facing ingress I ran into unexpected templating that differed between the two.

Describe the solution you'd like

Standardize and use default k8s Ingress object key

Describe alternatives you've considered

Additional context

Bump Honeycomb

Update Honeycomb to latest version

Bump Refinery

Update Refinery to latest version

#146 described a problem, and #155 fixed it, but only for the collector helm chart. We should have the same feature in the Honeycomb helm chart.

The helm chart references agent with tag 2.6.0, yet it does not exist:
https://hub.docker.com/r/honeycombio/honeycomb-kubernetes-agent/tags

(I'm assuming the travis job overrode the previous 2.5.6 tag, instead of pushing the new 2.6.0 one)

Is your feature request related to a problem? Please describe.
The HPA templates are using the deprecated autoscaling/v2beta1 and autoscaling/v2beta2 versions. Ideally these would be updated to autoscaling/v2.

Describe the solution you'd like
Update the templates the HPA templates to the autoscaling/v2 format. Very minimal changes are required to do this.

Describe alternatives you've considered
Could potentially introduce a new variable in the values file to allow using an older version of the HPA resource if further backwards compatability is required.

Additional context
Happy to PR in either approach if that would be helpful.

Is your feature request related to a problem? Please describe.

Internally we have an ingress controller based on nginx. To use that feature we simply configure our values.yaml this way:

ingress:
--
  | enabled: true
  | ingressClassName: https-internal
  | hosts:
  | - host: foo.bar.com
  | paths: ["/"]
  | tls: []

However, the ingress did not work properly. My colleague pointed out that ingressClassName was missing in the manifest spec. This feature was added a while back.

spec:
  rules:
    - host: foo.bar.com
      http:
        paths:
          - backend:
              service:
                name: secure-tenancy
                port:
                  name: http
            path: /
            pathType: Prefix

Describe the solution you'd like

The spec should have been this way:

spec:
  ingressClassName: https-internal
  rules:
...

Describe alternatives you've considered
N/A

Additional context

Is your feature request related to a problem? Please describe.
HPA objects incorrectly show as out of sync in ArgoCD if the metrics ordering doesn't match what's expected. This has been resolved in the opentelemetry-collector chart (see here: #164), but is still present in the refinery and secure-tenancy charts.

Describe the solution you'd like
Reorder the resouce metrics in the refinery and secure-tenancy charts to have memory before cpu.

Additional context
Happy to PR this in if you agree with it.

Is your feature request related to a problem? Please describe.

We utilize the helm templating engine to generate our manifests which we then check in and apply via kubectl at a later time. Right now, this chart doesn't inject the helm release namespace into the resources' metadata. It instead relies on the underlying kubectl client having a namespace flag specified.

Describe the solution you'd like

A common pattern is to include this in the templates

metadata:
    namespace: {{ .Release.Namespace }}

This lets you still use the existing way of applying via helm or kubectl and passing the namespace flag, but also allows generating templates with the namespace dictated in the yaml.

Describe alternatives you've considered

We could work around this issue by writing code so our deployment pipeline can take the namespace and apply it with kubectl. Other charts we use specify this namespace, so it would be nice if they were consistent.

Additional context
There was a long conversation about this here. It was ultimately closed as they won't dictate one way or the other. There are lots of issues that link to it though where others have added in this config.

Let me know your thoughts on this particular matter. If you approve, we can do the PR. Thanks for your consideration!

Is your feature request related to a problem? Please describe.
Currently use the app of apps model with argocd to deploy applications across a number of clusters which allows core components to be deployed on a number of clusters. Often times when in development, we want to isolate deployment of an application to a single dev cluster. For helm charts we've written, we can modify the template to include an enabled flag i.e {{- if .Values.my-app-chart.enabled }} and thus set enabled to false for a Values file of higher precedence. However, refinery is a dependency for a larger parent chart and thus we don't have the template yaml, and can't make minute modifications to an upstream chart, in this case refinery.

Describe the solution you'd like
An enabled flag for the entire chart.

Describe alternatives you've considered
Setting replicaCount to 0, but I think we'll still end up with other resources like ConfigMaps,Services etc being created unecessarily.

Additional context

Is your feature request related to a problem? Please describe.

The OpenTelemetry Collector chart doesn't support as many deployment options as the official one.

Describe the solution you'd like

Have this chart be a subchart of the official chart.

Describe alternatives you've considered

Additional context

The default service port for redis is called http which causes Istio to auto discover the protocol as HTTP, causing issues.

Jaeger uses UDP for thrift binary and thrift compact transports. To send data to opentelemetry-collector via jaeger we need to open up UDP instead of TCP.

I've created a PR for this: #46

https://github.com/honeycombio/refinery/releases/tag/v1.11.0

Tracking issue for chart update to support the latest ST release.

I currently can not set the priority of the associated honeycomb pods that I have in my Kubernetes cluster, because I lack the ability to set them via passing it in as a value.

Can you please add this ability in your next release?

An example of this in action

Thanks.

Bump Honeycomb Kubernetes Agent

Update Honeycomb Kubernetes Agent to latest version

Bump Refinery

Update Refinery to latest version

We saw an issue recently with the Helm chart for Refinery where adding some new hashing fields to the config didn't trigger a restart, and resulted in weird behavior from the cluster that forced us to manually restart the Refinery pods.

Versions

Chart: 1.3.0
Helm: ArgoCD v2.0.3+8
Kubernetes cluster: EKS 1.18.20
Refinery Version: "v1.6.1@sha256:e742af48b2ac1eb6b0116246780eb7e4942f0251152742990def677d853a129a"

Steps to reproduce

1. Add a new field to the values.yml override for hashing field config
2. Deploy the new config
3. Notice that Refinery pods do not restart to consume the new config

1.1.2 (+?)

Description

https://artifacthub.io/packages/helm/honeycomb/refinery/1.1.2?modal=security-report

CC @puckpuck

Is your feature request related to a problem? Please describe.
Bringing this here for discussion at @bdarfler's request.

I made a change today to the values we specify for our lacework deploy in a particular environment, and on my first attempt, got it wrong.

However, this got caught in our dry run in CI, because lacework's chart contains a values.schema.json, and so the pre-merge dry-run was able to tell me:

Error: UPGRADE FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
lacework-agent:
- image.imagePullSecrets.0: Invalid type. Expected: object, given: string

Which is pretty great! I think we may want to do similar work for our helm charts such that helm upgrade --install --dry-run can tell a developer "hi you've written a values.yml for your config that isn't valid". Like having TypeScript annotations for a library.

It may be helpful to have a sign-up link in the readmes for these charts. It doesn't seem inconceivable that someone might be interested in honeycomb, see if there's a helm chart published in a repository (e.g., artifacthub), and install it and then go from there. If there's a link to create a free account, that lets them do one less google query.

Versions

Chart: Refinery
Helm: 3.2
Kubernetes cluster: 1.22

Steps to reproduce

1. Set podLabels
2. Install chart

Additional context
Need to use podLabels to set a label that Istio recognizes for sidecar auto-injection.

The Honeycomb opentelemetry-collector chart was created before a community opentelemetry-collector chart existed and filled the important need to easily deploy a collector that could export data to Honeycomb.

Today there is an OpenTelemetry Community Collector helm chart that is actively maintained and supports more features than our chart. In addition, exporting data to Honeycomb via the OpenTelemetry Collector has been simplified to the OTLP exporter with a small amount of configuration. At this time the only "advantage" our chart has is the ability to configure the OTLP exporter based on the user-supplied API key and dataset name. But the advantage comes at the cost of lack of useful k8s features, automatic complex component support (see the community chart's preset concept for more details`), and regular updates.

I propose we deprecate our opentelemetry-collector chart in favor of the community opentelemetry-collector chart, supplementing the Honeycomb-specific configuration that our chart provided with Honeycomb documentation. Instead of investing our time in our own chart, we invest our time helping maintain the OpenTelemetry community chart.

Is your feature request related to a problem? Please describe.

The latest (1.1.0) uses 0.43.0, however Latest OTEL collector release is 0.52.0

Additional context Please note this request is being made on behalf of our customer. Obtaining additional context.

Is your feature request related to a problem? Please describe.

There is no information at https://honeycombio.github.io/helm-charts which is confusing for those that navigate to it.

Describe the solution you'd like

Add some content that steers users in a better direction.

Describe alternatives you've considered

Additional context

Versions

Application version 1.15.0
Chart versions 1.12.0 (17 Aug, 2022)

Steps to reproduce

Original issue has been fixed but latest chart does not appear to be released on public repo:
https://artifacthub.io/packages/helm/honeycomb/refinery

Please see my comment on pull request which has been merged.

Is your feature request related to a problem? Please describe.
I've seen behavior from the Refinery chart where pods will be scheduled on the same Kubernetes Node

Describe the solution you'd like
We've seen good success using topologySpreadConstraints to spread workload across K8s nodes using a topologyKey like topology.kubernetes.io/zone. Allowing operators to choose a topologyKey would allow us to mandate a more fault-tolerant HA topology.

Is your feature request related to a problem? Please describe.

Currently there is no automated test execution for PRs. As a result, we must rely on manual tests to ensure changes are accurate.

• add new workflow
• add CI scenarios to honeycomb
• add CI scenarios to opentelemetry-collector
• add CI scenarios to refinery

Describe the solution you'd like

Add a circleci workflow that uses chart-testing to test chart scenarios.

I'm comparing the line above to the docs: https://docs.honeycomb.io/manage-data-volume/refinery/scale-and-troubleshoot/#xalive

Bump Refinery

Update Refinery to latest version

Bump Refinery

Update Refinery to latest version

Is your feature request related to a problem? Please describe.
I've seen the tuning for MaxAlloc and CacheCapacity cause some confusion for operators of Refinery in Kubernetes.

Since sane default configs are documented here, why not leverage Helm to configure these?

Describe the solution you'd like
Since Refinery pods should get Guaranteed QoS in K8s anyway (requests and limits on RAM should be matching values), we could use those RAM allocation values to calculate the default values for both MaxAlloc and CacheCapacity. I think we'd want these to be overridable, however, via something like a custom values.yaml file.

Describe alternatives you've considered
I'm considering opening a PR on this myself, but need to negotiate for the time to do so from $job.

A few complaints in the field about Refinery pod cycling many times before getting stable. There is some thought that a readinessProbe may help mitigate the effect.

https://circleci.com/blog/ubuntu-14-16-image-deprecation/

Received notice that a job uses an Ubuntu 14.04-based image.

The line machine: true should change to specify a specific version > 16.04

Bump Refinery

Update Refinery to latest version

Update ingress spec to work with Kubernetes 1.22.

In K8s 1.19 official support for the ingress API was added. In 1.22 support for the beta API was removed.

Add the additionalFields top level config object, as well as any other missing configuration items to the Honeycomb K8s agent chart.

Bump Refinery

Update Refinery to latest version

#146 described a problem, and #155 fixed it, but only for the collector helm chart. We should have the same feature in the Refinery helm chart.

I want to prevent refinery pods from being scheduled on the same node, but allow them to be scheduled on the same node as the refinery-redis pod. I have this right now:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - topologyKey: "kubernetes.io/hostname"
        labelSelector:
          matchExpressions:
            - key: "app.kubernetes.io/name"
              operator: In
              values: [refinery]

But this doesn't work because it applies to both the refinery and refinery-redis deployments, so the refinery-redis pod doesn't allow any refinery pods to be scheduled on its node. Is there a workaround here I'm missing, or would this require splitting out the affinities in the values file? I'm pretty new to k8s affinities.

Thanks!

Update ingress spec to work with Kubernetes 1.22.

In K8s 1.19 official support for the ingress API was added. In 1.22 support for the beta API was removed.

The refinery helm chart hard codes the deployment port values. This means changes to the ListenAddr, GRPCListenAddress, or service ports will not work as expected.

We should change these lines to honor service port values changes: