Reduce krakenFailThreshold? about calamari-clients HOT 3 OPEN

@tserong I think we can probably tighten this up. cthulhu schedules heartbeats to run ever 10s. So the lower-bound for new heartbeat is 10s + processing time.
In the GUI this window is dependent on cluster_update_time_unix being different from Date.now();
both expressed as ms since epoch.

The cluster update time is sourced from the health sync_object which is subject to the favorite window, which in some cases, can be as high as 3-4 minutes. If things are working as expected even the favorite window should only be 30-40 seconds.

I would imagine that we could easily change this to 5min. We could try 1-2min and see what happens. What do you think?

from calamari-clients.

@GregMeno apologies for the laggy reply.

If the favorite window gets as high as 3-4 minutes, is that indicative of some sort of problem? Because if so, I'd lean towards setting the fail threshold to 1-2min anyway, so that case becomes obvious (or obvious-ish). But if 3-4 minutes is expected sometimes, and isn't scary, then I guess 5min would be more sensible for the fail threshold.

from calamari-clients.

Ping?

from calamari-clients.