Reverse proxy startup health check behavior results in 503 errors about caddy HOT 5 OPEN

Note that health_passes 3 means that after failing, an upstream node needs to pass three successive health checks to become healthy again.

I'm okay with 1 as long as the current behavior remains where a health check is immediately fired and the block is near instantaneous.

I believe other loadbalancers like Nginx (paid) assumes that all listed upstreams are healthy after a reload/restart and doesn't take them out of the mix until the health checks fail.

from caddy.

Marking them as healthy when you don't actually know seems... misleading?

i think the argument can be made that marking them unhealthy is equally misleading. the remote is in superposition, since it has not been obvserved, it's a third distinct state that is currently handled as the unhealthy case. it seems existing implementations tip the scale slightly in favor the healthy superposition, my guess is it is order to have a faster time to first response.

from caddy.

(As clarified in Slack, this is about active Health checks.)

My vote is currently for no. 1.

2 - Is a NO from me because blocking provisioning can make config reloads slow, and we strive to keep them fast and lightweight.

3 - is a NO from me because if the proxy is started before the backends, we can't assume that backends are healthy right away. IMO, active health checks should assume unhealthy unless proven otherwise by a passing health check (compared to passive health checks, which assume healthy until proved otherwise).

Number 1 is nice because it allows the server/config to start quickly, and the requests don't have to fail (even if they are delayed briefly). We also don't have bad status information. I imagine health checks -- especially passing ones -- happen very quickly, so the blocking will be instantaneous, less than 1/4 second probably.

from caddy.

I believe other loadbalancers like Nginx (paid) assumes that all listed upstreams are healthy after a reload/restart and doesn't take them out of the mix until the health checks fail.

basically correct. during investigation i found that nginx plus and traefik set the initial state of the backend when no health checks have been made to them as healthy. However, they do preserve history across restarts to the same hosts (as does caddy, i believe)

from caddy.

basically correct. during investigation i found that nginx plus and traefik set the initial state of the backend when no health checks have been made to them as healthy.

I didn't think about what other servers do when we implemented health checks, but this is surprising to me... it feels wrong for active health checks to assume a healthy backend without checking first. Marking them as healthy when you don't actually know seems... misleading?

However, they do preserve history across restarts to the same hosts (as does caddy, i believe)

Caddy preserves the health status across reloads but if the process quits then the memory is cleared. We don't persist it to storage as of yet.

from caddy.