Prometheus data source for "Azure Monitor for Prometheus" with workload identity does not work about grafana-operator HOT 5 CLOSED

@sirish-bajpai

1. The operator does not need to support Workload Identity in this scenario, because it interacts only with Grafana, not with the datasource.
2. I see that you used a wrong data source type (type: prometheus). Based on the documentation you shared, it has to be set to type: grafana-azure-monitor-datasource, so your updated example would look like this:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: azure-monitor-prometheus
  namespace: grafana
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  datasource:
    name: "azure-monitor-prometheus"
    type: grafana-azure-monitor-datasource # THIS
    access: proxy
    editable: true
    url: "https://xxxxxxxxx.eastus2.prometheus.monitor.azure.com"
    jsonData:
      manageAlerts: true
      cacheLevel: "High"
      disableRecordingRules: false
      incrementalQueryOverlapWindow: "10m"
      azureAuthType: "workloadidentity"

Also, make sure workload identity is enabled in your Grafana instance:

spec:
  config:
    azure:
      workload_identity_enabled: "true"

NOTE: if client ID and tenant ID are not provided through respective environment variables, you'll need to manually pass them through workload_identity_tenant_id and workload_identity_client_id in the same configuration section. - It all is described in the documentation.

I hope that helps.

P.S. Please, use code blocks when pasting yaml to make sure indentation is preserved, otherwise we cannot copy your examples.

from grafana-operator.

@weisdd
Thanks very much for your comment.
Unfortunately though, not the right solution.
Under the "Azure Monitor" umbrella, Azure packages a Prometheus data service.
Prometheus metrics can be injected in this service from Azure Kubernetes nodes and pods.
Other Azure monitoring datum like azure metrics, traces, logs and resource graph can be accessed via the 'grafana-azure-monitor-datasource', but not the prometheus metrics. For that, Azure provides a specific URL (notice the .prometheus.. in the URL), which needs to be configured as 'prometheus' datasource. Here is their documentation:
https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/prometheus-grafana
Now, when I create this prometheus datasource from grafana UI, I can see the Azure auth option, and Workload Identity:

And from the 'explore data' view, I can see all the scraped metrics. So the auth works from UI.
But not from operator. I mentioned the 'Azure Monitor' in my post as an example, assuming the same provisioning json fields should work for prometheus service also.
Hope this clarifies the scenario.

from grafana-operator.

@sirish-bajpai Alright, I think I can help you. :)

Basically, both the operator and Grafana UI send requests to the same backend API, so, in theory, you should be able to achieve the same configuration state.
When documentation is not clear on how to configure something, you can always dump POST/PATCH/PUT requests through Chrome DevTools ("Network" section).

What I can see in the dump is that the UI in Grafana 10.3.1 contains two settings related to workload identity:

  "jsonData": {
    "azureAuthType": "workloadidentity",
    "azureCredentials": {
      "authType": "workloadidentity"
    }
  }

When I ran some tests, it seemed like only the latter had any effect. Not sure if behaviour would be the same in other versions, so I would suggest you to use the same settings in GrafanaDatasource:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: azure-monitor-prometheus
  namespace: grafana
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  datasource:
    name: "azure-monitor-prometheus"
    type: prometheus
    access: proxy
    editable: true
    url: "https://xxxxxxxxx.eastus2.prometheus.monitor.azure.com"
    jsonData:
      manageAlerts: true
      cacheLevel: "High"
      disableRecordingRules: false
      incrementalQueryOverlapWindow: "10m"
      azureAuthType: "workloadidentity"
      azureCredentials:
        authType: "workloadidentity"

from grafana-operator.

@weisdd
That worked !!
Thanks very much.
I will make sure to take a look at Chrome dev next time I get stuck.

from grafana-operator.

@sirish-bajpai no worries, was happy to help :)

from grafana-operator.