Better stemcell alerts about prometheus-boshrelease HOT 9 CLOSED

There's a third approach (or complementary to the above). Instead of returning 0 or 1 we can return when the stemcell was released (in unix timestamp format). From here we can match each deployment stemcell with the release date and emit an alert if it's older than x sec/mins/days (time() - (bosh_deployment_stemcell_info * on(bosh_stemcell_version) group_left bosh_stemcell_info) > x).

w/r/t the stemcell series, one thing we can do is to emit also labels for the major and minor versions, both at the bosh_io_exporter and the bosh_exporter. This will allow us to match the exact stemcell series and alert on minor versions.

Oh, and we can do the same for releases 😛.

from prometheus-boshrelease.

I don't think I care how old a stemcell is in time--I only care whether it's the latest version. If no CVEs get patched over a few weeks it's fine that stemcells are a few weeks old, but if there are a few stemcell updates over a week, allowing stemcells to be a few weeks old would be bad.

I'll think about this more and hopefully write a quick bosh-io-exporter over the weekend.

from prometheus-boshrelease.

Let me rephrase it. There're customers that don't live on the edge, and they update stemcells ie once a week. Having an alert that notifies them they don't have the latest stemcell they day after the stemcell has been released doesn't make any sense to them, they know they risk and accept it. Instead, having an alert that notifies them that the stemcells they're using are outdated if the latest one has been released more than 1 week ago make sense, hence that adding the timestamp makes sense. Also I find more intuitive to use a timestamp as value that not 0 or 1.

from prometheus-boshrelease.

A few things:

• It doesn't look like the bosh-hub api currently includes release timestamps, although that would be a quick patch to https://github.com/cppforlife/bosh-hub/blob/a745e1693e553b5a5a2b07cf76dfb82be8344a93/ui/stemcell/stemcell.go#L224-L234
• I think I'm still misunderstanding how you're suggesting the alert would work. As far as I can tell, stemcell releases don't have a regular cadence but happen when cves get patched. Let's say a new stemcell doesn't come out for two weeks, and I configure my alerts to go off after a week. It sounds like I would start getting spurious alerts after a week, which isn't useful--I never want to get alerted if I'm using the current stemcell. Do you mean that we would alert if the stemcell is older than some interval unless it's the latest in the series?

Thanks / sorry if I'm misunderstanding!

from prometheus-boshrelease.

No, what I mean is "Alert only if a deployment is not using the latest stemcell in the series and it has been more than x time since it was released". What I want to avoid is to emit alerts if I'm not using the latest one and that one was released yesterday (because I'm upgrading stemcell once a week). I guess this is the same as using the "FOR" clause, but using an alert rule (to not have a pending alert during the FOR time, that can be pretty large).

from prometheus-boshrelease.

When you say...

"Alert only if a deployment is not using the latest stemcell in the series and it has been more than x time since it was released"

...do you mean time since the outdated stemcell was released, or time since the current stemcell was released? Based on the situation you're describing, I'm guessing you mean time since current stemcell, since you're trying to avoid annoying alerts right after a stemcell comes out.

It sounds like we would have metrics like this:

bosh_stemcell_info{bosh_stemcell_version="3312.29",current="true"} 1499047468
bosh_stemcell_info{bosh_stemcell_version="3312.28",current="false"} 1499047468
bosh_stemcell_info{bosh_stemcell_version="3312.27",current="false"} 1499047468

...and a query like this:

time() - (bosh_deployment_stemcell_info * on(bosh_stemcell_version) group_left bosh_stemcell_info{current="false"}) > x

Is that what you had in mind?

from prometheus-boshrelease.

Yep, something like that will work.

from prometheus-boshrelease.

This issue is stale because it has been open 60 days with no activity. Comment or this will be closed in 5 days.

from prometheus-boshrelease.

This issue was automatically closed because it has been stalled for 5 days with no activity.

from prometheus-boshrelease.