Alerting on missing processes upon startup about process-exporter HOT 6 OPEN

Thanks for the detailed response @ncabatoff - makes total sense.

num_procs pre-populated with a value of zero even if no processes are found.

This would be the most ideal solution for me, but I understand it's extra work. I'd be grateful if you'd consider this enhancement for a future release. Thanks!

from process-exporter.

We could use an up like metric for processes

from process-exporter.

Very happy to hear that you find it useful. I don't think you're doing anything wrong. If I understand your problem correctly it's a classic issue with Prometheus-style monitoring. Unless Prometheus has seen a given metric recently, it doesn't know about it. Thus the alert rule never fires initially, because namedprocess_namegroup_num_procs is not < 1: it's undefined.

If you know the names of the namegroups you care about, you could also check for their absence, e.g.

expr: namedprocess_namegroup_num_procs{groupname="X"} < 1 or absent(namedprocess_namegroup_num_procs{groupname="X"})

That's still not quite enough, because for a binary operation like or to work the labelsets must match exactly on either side. One easy way to make that work (by eliminating labels) is to use an aggregation like sum(), e.g.

expr: sum(namedprocess_namegroup_num_procs{groupname="X"}) < 1 or absent(namedprocess_namegroup_num_procs{groupname="X"})

There are other less hacky options like ignoring or on, see the operators page for details.

If you don't want to hardcode the names of the namegroups you care about, I don't have any good suggestions with the existing process-exporter, but if you think of any I'd love to hear them.

One possibility might be extend process-exporter so that any names given explicitly (as -procnames or with untemplated "name" values in the config file) would get num_procs pre-populated with a value of zero even if no processes are found. I'm going to leave this issue open, I might come back and implement that at some point.

from process-exporter.

Understood @geekdave, I'll see what I can do. I'm very open to adding this feature, just have other more pressing issues at the moment.

from process-exporter.

Prepopulating with zero using a runtime flag or config option would be great. Any progress on that yet?

from process-exporter.

Any update on this?

from process-exporter.