promhippie / prometheus-vcd-sd Goto Github PK

The HTTP SD within Prometheus accepts only application/json as a content type, so we need to drop the utf-8 encoding part from the content type.

Hi, Thank you for the great work, this is a useful tool.
We would like to use fixed tags in our deployment, but the 0.4.0 doesn't include #142 which affects our systems.
Can you create a new release tag?

We got to drop the darwin/386 builds, they are not supported by the used Go version anymore. Soonish we should also add arm64 builds to stay compatible with current macOS hardware.

It would be nice if a change to the configuration automatically reloads the service.

Since Prometheus implements a HTTP SD with the new version 2.28 we should provide an endpoint for this implementation as an alternative to the regular file handler.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: .github/renovate.json
Error type: The renovate configuration file contains some invalid settings
Message: Invalid configuration option: dockerfile.enableMajor

Hi,
We found you project very interesting but is it possible to add the esx host information on service discovery ?
The json output file doesn't include this interesting information and we would like to supervise when a node move to an other esx host.

Thanks a lot

To have a unified project structure accross the whole promhippie GitHub organization all projects should impelent a similar structure.

Target labels should contain the VM ID as provided by VCD.

E.g.: "__meta_vcd_id": "urn:vcloud:vm:807b799e-c72f-4592-9acd-ccefefe92720"

Hi, while using VCD-SD exporter, find this bug.

Discover service from prometheus-vcd-sd

 {
        "targets": [
            "10.12.146.50"
        ],
        "labels": {
            "__address__": "10.12.146.50",
            "__meta_vcd_id": "urn:vcloud:vm:02217ed9-ea98-495e-a577-d1a517017611",
            "__meta_vcd_name": "k8sw-dev-cubone",
            "__meta_vcd_network_k8s_net": "10.12.146.50",
            "__meta_vcd_num_cores_per_socket": "2",
            "__meta_vcd_num_cpus": "8",
            "__meta_vcd_org": "VDC",
            "__meta_vcd_os_type": "ubuntu64Guest",
            "__meta_vcd_project": "default",
            "__meta_vcd_status": "4",
            "__meta_vcd_storage_profile": "NonSSD",
            "__meta_vcd_vdc": "VDC"
        }
    },
    {
        "targets": [
            "10.12.147.4"
        ],
        "labels": {
            "__address__": "10.12.147.4",
            "__meta_vcd_id": "urn:vcloud:vm:12533e37-9ba1-4cd0-8919-d0360b6acdcf",
            "__meta_vcd_metadata_vmware.cloud.director.ui.badges.blue": "false",
            "__meta_vcd_metadata_vmware.cloud.director.ui.badges.green": "false",
            "__meta_vcd_metadata_vmware.cloud.director.ui.badges.magenta": "false",
            "__meta_vcd_metadata_vmware.cloud.director.ui.badges.orange": "false",
            "__meta_vcd_metadata_vmware.cloud.director.ui.badges.red": "false",
            "__meta_vcd_name": "jump-host",
            "__meta_vcd_network_srv_net": "10.12.147.4",
            "__meta_vcd_num_cores_per_socket": "1",
            "__meta_vcd_num_cpus": "2",
            "__meta_vcd_org": "VDC",
            "__meta_vcd_os_type": "ubuntu64Guest",
            "__meta_vcd_project": "default",
            "__meta_vcd_status": "4",
            "__meta_vcd_storage_profile": "NonSSD",
            "__meta_vcd_vdc": "VDC"
        }
    },

After this in log Prometheus contain error

prometheus  | ts=2022-02-15T09:44:43.523Z caller=refresh.go:79 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="\"__meta_vcd_metadata_vmware.cloud.director.ui.badges.green\" is not a valid label name"

docker-compose.yml

version: '2'

volumes:
  prometheus:

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: always
    ports:
      - 9090:9090
    volumes:
      - prometheus:/prometheus
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./service-discovery:/etc/sd

  vcd-exporter:
    image: promhippie/prometheus-vcd-sd:latest
    container_name: vcd-sd
    restart: always
    ports:
      - 9000:9000
    environment:
      - PROMETHEUS_VCD_LOG_PRETTY=true
      - PROMETHEUS_VCD_OUTPUT_ENGINE=http
      - PROMETHEUS_VCD_OUTPUT_FILE=/etc/sd/vcd.json
      - PROMETHEUS_VCD_URL=API_URL
      - PROMETHEUS_VCD_USERNAME=USER
      - PROMETHEUS_VCD_PASSWORD=PASS
      - PROMETHEUS_VCD_ORG=VDC
      - PROMETHEUS_VCD_VDC=VDC
    volumes:
      - ./service-discovery:/etc/sd

prometheus.yml

global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m

scrape_configs:
- job_name: node
  http_sd_configs:
  - url: http://vcd-sd:9000/sd

  relabel_configs:
  - source_labels: [__meta_vcd_network_internal]
    replacement: "${1}:9100"
    target_label: __address__
  - source_labels: [__meta_vcd_name]
    target_label: instance

docker version: 20.10.12
docker compose version: v2.2.3
OS: Debian GNU/Linux 11 (bullseye)

While the service is running fine it could happen after some amount of time that the authentication gets invalidaded, that would require to restart the service as it doesn't try to re-authenticate. Maybe it would make sense to authenticate on every loop and disconnect at the end of it.

Instead of adding development tooling to the regular go.mod file we should use bingo to version used tools like calens, golint, reflex and staticcheck to make sure we always use versions known to work properly.

For Prometheus there is a new toolkit available which enables a web config to secure exporters (or HTTP service discoveries) be integrating certs and also authentication. This should be implemented. You can find it at https://github.com/prometheus/exporter-toolkit.

If a vApp does not provide any VM currently it fails to process any service discovery as it panics. We got to check if a vApp provides any VM before looping through it.

Just prepared an initial basic version which could be released to the public.

Hi :)
I'm getting this error :
go:337 level=error component="discovery manager scrape" discovery=file msg="Error reading file" path=/etc/sd/vcd-vdcplus.json err="\"__meta_vcd_network_network name\" is not a valid label name"

It seems that a space in any label name is not good for Prometheus.
I'm not confident in developing in Go but if you don't have time I can take a look at it :)

Every network attached to the VMs gets transformed into a label, but currently there is no normalization of the network names. A network name like FOO-bar-1000 should be translated to __meta_vcd_network_foo_bar_1000, otherwise it's an invalid label.

Hi,

Thanks for this wonderful project not many project at github that can combine vcloud director metrics into prometheus.
But i still wondering how to use it? since, my prometheus can't read any information from output.log.
It can only read data like this. There's no info in output.log such as __meta_vcd_metadata_apps, __meta_vcd_num_cores_per_socket that listed in vcd-sd result page (http://localhost:9000/metrics)

are there any possibilities to read whatever in output.log said into Prometheus, coz output.log data is accurate but somehow can't be injected into vcd-sd metrics data (http://localhost:9000/metrics)

Thanks

We need to update the used Go version and on the same run we should also update all dependencies to the latest version to keep everything properly up to date.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository