MongoDB aggregation query exporter for Prometheus.
- Support for gauge metrics
- Pull and Push (Push is only supported for MongoDB >= 3.6)
- Supports multiple MongoDB servers
- Metric caching support
Note that this is not designed to be a replacement for the MongoDB exporter to instrument MongoDB internals. This application exports custom MongoDB metrics in the prometheus format based on the queries (aggregations) you want.
Get Prometheus MongoDB aggregation query exporter, either as a binaray from the latest release or packaged as a Docker image.
For kubernetes users there is an official helm chart for the MongoDB query exporter. Please read the installation instructions here.
You can run the exporter using docker (This will start it using the example config provided in the example folder):
docker run -e MDBEXPORTER_CONFIG=/config/configv3.yaml -v $(pwd)/example:/config ghcr.io/raffis/mongodb-query-exporter:latest$ mongodb-query-exporter
Use the -help flag to get help information.
If you use MongoDB Authorization, best practices is to create a dedicated readonly user with access to all databases/collections required:
- Create a user with 'read' on your database, like the following (replace username/password/db!):
db.getSiblingDB("admin").createUser({
user: "mongodb_query_exporter",
pwd: "secret",
roles: [
{ role: "read", db: "mydb" }
]
})- Set environment variable
MONGODB_URIbefore starting the exporter:
export MDBEXPORTER_MONGODB_URI=mongodb://mongodb_query_exporter:secret@localhost:27017Note: The URI is substituted using env variables ${MY_ENV}, given that you may also pass credentials from other env variables. See the example bellow.
If you use x.509 Certificates to Authenticate Clients, pass in username and authMechanism via connection options to the MongoDB uri. Eg:
mongodb://CN=myName,OU=myOrgUnit,O=myOrg,L=myLocality,ST=myState,C=myCountry@localhost:27017/?authMechanism=MONGODB-X509
You can pass in credentials from env variables.
Given the following URI the exporter will look for the ENV variables called MY_USERNAME and MY_PASSWORD and automatically use them at the referenced position within the URI.
export MY_USERNAME=mongodb_query_exporter
export MY_PASSWORD=secret
export MDBEXPORTER_MONGODB_URI=mongodb://${MY_USERNAME}:${MY_PASSWORD}@localhost:27017The metrics are by default exposed at /metrics.
curl localhost:9412/metrics
The exporter is looking for a configuration in ~/.mongodb_query_exporter/config.yaml and /etc/mongodb_query_exporter/config.yaml or if set the path from the env MDBEXPORTER_CONFIG.
You may also use env variables to configure the exporter:
| Env variable | Description | Default |
|---|---|---|
| MDBEXPORTER_CONFIG | Custom path for the configuration | ~/.mongodb_query_exporter/config.yaml or /etc/mongodb_query_exporter/config.yaml |
| MDBEXPORTER_MONGODB_URI | The MongoDB connection URI | mongodb://localhost:27017 |
| MDBEXPORTER_MONGODB_QUERY_TIMEOUT | Timeout until a MongoDB operations gets aborted | 10 |
| MDBEXPORTER_LOG_LEVEL | Log level | warning |
| MDBEXPORTER_LOG_ENCODING | Log format | json |
| MDBEXPORTER_BIND | Bind address for the HTTP server | :9412 |
| MDBEXPORTER_METRICSPATH | Metrics endpoint | /metrics |
Note if you have multiple MongoDB servers you can inject an env variable for each instead using MDBEXPORTER_MONGODB_URI:
MDBEXPORTER_SERVER_0_MONGODB_URI=mongodb://srv1:27017MDBEXPORTER_SERVER_1_MONGODB_URI=mongodb://srv2:27017- ...
Since the v1.0.0 release you should use the config version v3.0 to profit from the latest features. See the configuration version matrix bellow.
Example:
version: 3.0
bind: 0.0.0.0:9412
log:
encoding: json
level: info
development: false
disableCaller: false
global:
queryTimeout: 3s
maxConnection: 3
defaultCache: 0
servers:
- name: main
uri: mongodb://localhost:27017
aggregations:
- database: mydb
collection: objects
servers: [main] #Can also be empty, if empty the metric will be used for every server defined
metrics:
- name: myapp_example_simplevalue_total
type: gauge #Can also be empty, the default is gauge
help: 'Simple gauge metric'
value: total
overrideEmpty: true # if an empty result set is returned..
emptyValue: 0 # create a metric with value 0
labels: []
constLabels:
region: eu-central-1
cache: 0
mode: pull
pipeline: |
[
{"$count":"total"}
]
- database: mydb
collection: queue
metrics:
- name: myapp_example_processes_total
type: gauge
help: 'The total number of processes in a job queue'
value: total
labels: [type,status]
constLabels: {}
mode: pull
pipeline: |
[
{"$group": {
"_id":{"status":"$status","name":"$class"},
"total":{"$sum":1}
}},
{"$project":{
"_id":0,
"type":"$_id.name",
"total":"$total",
"status": {
"$switch": {
"branches": [
{ "case": { "$eq": ["$_id.status", 0] }, "then": "waiting" },
{ "case": { "$eq": ["$_id.status", 1] }, "then": "postponed" },
{ "case": { "$eq": ["$_id.status", 2] }, "then": "processing" },
{ "case": { "$eq": ["$_id.status", 3] }, "then": "done" },
{ "case": { "$eq": ["$_id.status", 4] }, "then": "failed" },
{ "case": { "$eq": ["$_id.status", 5] }, "then": "canceled" },
{ "case": { "$eq": ["$_id.status", 6] }, "then": "timeout" }
],
"default": "unknown"
}}
}}
]
- database: mydb
collection: events
metrics:
- name: myapp_events_total
type: gauge
help: 'The total number of events (created 1h ago or newer)'
value: count
labels: [type]
constLabels: {}
mode: pull
# Note $$NOW is only supported in MongoDB >= 4.2
pipeline: |
[
{ "$sort": { "created": -1 }},
{"$limit": 100000},
{"$match":{
"$expr": {
"$gte": [
"$created",
{
"$subtract": ["$$NOW", 3600000]
}
]
}
}},
{"$group": {
"_id":{"type":"$type"},
"count":{"$sum":1}
}},
{"$project":{
"_id":0,
"type":"$_id.type",
"count":"$count"
}}
]See more examples in the /example folder.
By defining no actual value field but set overrideEmpty to true a metric can sill be exported
with labels from the aggregation pipeline but the value is set to a static value taken from emptyValue.
This is useful for exporting info metrics which can later be used for join queries.
servers:
- name: main
uri: mongodb://localhost:27017
aggregations:
- database: mydb
collection: objects
metrics:
- name: myapp_info
help: 'Info metric'
overrideEmpty: true
emptyValue: 1
labels:
- mylabel1
- mylabel2
constLabels:
region: eu-central-1
cache: 0
mode: pull
pipeline: `...`| Config version | Supported since |
|---|---|
v3.0 |
v1.0.0 |
v2.0 |
v1.0.0-beta5 |
v1.0 |
v1.0.0-beta1 |
Prometheus is designed to scrape metrics. During each scrape the mongodb-query-exporter will evaluate all configured metrics. If you have expensive queries there is an option to cache the aggregation result by setting a cache ttl. However it is more effective to avoid cache and design good aggregation pipelines. In some cases a different scrape interval might also be a solution. For individual aggregations and/or MongoDB servers older than 3.6 it might still be a good option though.
A better approach is using push instead a static cache, see bellow.
Example:
aggregations:
- metrics:
- name: myapp_example_simplevalue_total
help: 'Simple gauge metric which is cached for 5min'
value: total
servers: [main]
mode: pull
cache: 5m
database: mydb
collection: objects
pipeline: |
[
{"$count":"total"}
]To reduce load on the MongoDB server (and also scrape time) there is a push mode. Push automatically caches the metric at scrape time preferred (If no cache ttl is set). However the cache for a metric with mode push will be invalidated automatically if anything changes within the configured MongoDB collection. Meaning the aggregation will only be executed if there have been changes during scrape intervals.
Note: This requires at least MongoDB 3.6.
Example:
aggregations:
- metrics:
- name: myapp_example_simplevalue_total
help: 'Simple gauge metric'
value: total
servers: [main]
# With the mode push the pipeline is only executed if a change occured on the collection called objects
mode: push
database: mydb
collection: objects
pipeline: |
[
{"$count":"total"}
]The mongodb-query-exporters also publishes a counter metric called mongodb_query_exporter_query_total which counts query results for each configured aggregation.
Furthermore you might increase the log level to get more insight.
- The balloon helm chart implements the mongodb-query-exporter to expose general stats from the MongoDB like the number of total nodes or files stored internally or externally. See the config-map here.
Please submit a PR if your project should be listed here!
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent