Health Checks and Failure Handling

Asking whether a machine is alive

A balancer still routing to a dead server is worse than no balancer, so it keeps asking.

Every few seconds each machine gets a probe, usually a request to /health that should come back with a 200. Miss a few in a row and the machine leaves the rotation. Pass a few in a row and it earns its way back.

Those two knobs are the whole story. because they are the whole story. How long you serve errors is roughly the interval times the number of failures you tolerate. Check every 10 seconds, allow 3 failures, and real people eat errors for up to 30 seconds. Tighten them and a brief pause on a healthy machine ejects it. Loosen them and outages linger.

What /health should check

Decide carefully what /health actually checks, because this is where teams take themselves down. A shallow check answers 200 if the process is running.

A deep check tries the database and the cache first. That feels more honest, and it is how you lose an entire fleet at once. The database stutters for 20 seconds, every machine fails its check together, the balancer ejects all of them, and a blip becomes a full outage that outlasts the blip.

Shallow checks belong on your balancer and deep dependency checks for your monitoring, which is the convention almost everyone lands on eventually.

Failure handling goes past detection. Draining lets a machine you are removing finish the requests it already has instead of cutting them off mid-response.

Some balancers also watch real traffic and pull a machine that starts throwing errors faster than the probe cycle would. And one that fails open, sending traffic everywhere when every single check fails, is making a sensible bet that the checks are lying. A whole fleet rarely dies at the same instant.

// Shallow: is this process alive and serving? That is all the
// balancer needs to know.
get("/health", (req, res) -> res.status(200).body("ok"));

// Deep: checks the database too. Looks more honest, and when the
// database blips for 20 seconds every machine fails together and
// the balancer ejects the entire fleet.
get("/health/deep", (req, res) -> {
  db.ping();                    // use this for monitoring,
  cache.ping();                 // never for the balancer
  return res.status(200);
});

// Detection lag = interval x threshold. 10s x 3 = up to 30 seconds
// of real users getting errors before the machine is pulled.

Worked example

Tariq gets paged at 3:40 am: the site is serving 503s from the ALB itself, no healthy targets. All 8 API nodes are running fine. The timeline reconstructs cleanly. RDS ran a 25-second failover at 3:31. The team's /health endpoint did a SELECT 1 against that database, so all 8 nodes failed 3 consecutive checks together and the ALB ejected the entire fleet. The database recovered at 3:32, but each node then needed 3 passing checks at 10-second intervals to be re-admitted, so the outage stretched to nearly 4 minutes for a 25-second blip. The fix: /health now returns 200 whenever the process can serve requests, and database connectivity moved to a CloudWatch alarm that pages a human instead of the balancer.