Skip to main content
Alertinglesson 3 of 4 · 3 min read

Killing Alert Fatigue

A trust problem with a body count

Alert fatigue is a trust problem, and its body count is the incidents you missed.

When your on-call gets paged nightly for things needing no action, acknowledging becomes a reflex performed half asleep. Your reflex cannot tell the fortieth false alarm from the real outage.

Recognise the pattern, which has a name from a very different field: each ignored alarm makes ignoring alarms feel safer. The endpoint is somebody sleeping through the page that mattered, and the correct response is to blame the alert load rather than the human.

Measure, then delete

Measure first. Track your pages per shift per person, and review every one weekly with a single question: what action did this produce?

Set a usefully low bar. More than two incidents in a twelve hour shift is overload worth escalating to management.

Give every action-free alert a verdict in that review. Delete it, demote it to a ticket, raise its threshold, or fix the flapping condition underneath it.

Delete without flinching, even though your first few will feel reckless. It is the opposite of reckless, because every deletion raises the signal value of everything that remains.

Take a layer of noise off mechanically too, because your alerting tools support grouping, so thirty instances of one database failure arrive as one incident instead of thirty pages.

Suppress your downstream alerts when the upstream cause is already firing, because you do not need twelve of them announcing that the database they share is down. And silence during planned work, which prevents the deploy-window storm that trains people to ignore Thursday afternoons.

Keep one cultural rule to hold the system honest over time. Anybody may propose deleting an alert, and its defender must name the concrete action it demands and the recent firing where somebody took that action.

No answer, no alert. Your pager exists to interrupt sleep for user-facing emergencies, and it has to be quiet enough that a page means exactly that.

the shape of it
30 alertsone database downGroupedone incidentDownstream mutedcause is firingOne page1. same root cause2. upstream known3. one human woken4. and only one
step 1 of 4
Grouping and inhibition are why one failure wakes one person, not thirty.

Worked example

An infrastructure team at an adtech company averages 340 pages a month across four on-call engineers, and it costs them a resignation: their strongest engineer quits citing two years of wrecked sleep, and the exit interview lands on the VP's desk. The remaining team declares bankruptcy on the alert config. For one month, every page gets a row in a spreadsheet: timestamp, alert, action taken. The numbers are damning, 82 percent of pages produced no action beyond acknowledgment, and 11 alerts accounted for two thirds of the noise. They delete 60 of 140 rules outright, convert 35 to tickets, add grouping and three inhibition rules, and route deploy-window noise into silences. Four months later the count is 24 pages a month, action rate 78 percent, and the next quarterly engagement survey shows on-call satisfaction up more than any other metric.