Possible bug, marked as successful health check when it should have timed out about cabot HOT 4 CLOSED

Unless the check in fact passed? How do you know it should have failed?

from cabot.

hey @dbuxton, thanks for looking into it.

The timeout is set to 30ms, so if a check crosses that threshold, it should be marked as a failure no? In the first check, it took 32ms & was marked as failure, while the second took 558ms but was successful.

from cabot.

Thanks for clarifying. We don't actually use the reported time to mark checks as successful or failing (instead we pass the value to the underlying call to requests so that it knows when to time out the request itself). I'm pretty sure that the times we're getting reported are actually totally wrong but the statuses are correct, if you see what I mean.

To give you some idea of how we use this ourselves, we're typically passing in an upper bound of when a service would be considered "up" (might be 10s or something high). If we want to alert on response times per se, we gather that information in graphite and then watch whatever metric we need to with cabot. The only reason we'll play with the timeout is if we have a really slow service that we nonetheless don't want to get alerted by all the time.

from cabot.

Cool. Thanks for clarifying. Great tool! Looking forward to contribute to
it at some point after being becoming more familiar. Cheers!
On Jun 13, 2014 4:12 PM, "dbuxton" [email protected] wrote:

Thanks for clarifying. We don't actually use the reported time to mark
checks as successful or failing (instead we pass the value to the
underlying call to requests so that it knows when to time out the request
itself). I'm pretty sure that the times we're getting reported are actually
totally wrong but the statuses are correct, if you see what I mean.

To give you some idea of how we use this ourselves, we're typically
passing in an upper bound of when a service would be considered "up" (might
be 10s or something high). If we want to alert on response times per se, we
gather that information in graphite and then watch whatever metric we need
to with cabot. The only reason we'll play with the timeout is if we have a
really slow service that we nonetheless don't want to get alerted by
all the time.

—
Reply to this email directly or view it on GitHub
#70 (comment).

from cabot.