Paging on causes
The most common alerting mistake is paging on causes. Processor over 80 percent, memory over 90, disk filling, a container restarting.
Each of those sounds responsible, and each fires constantly in situations your users never notice. A nightly batch job pins the processor. Your runtime sits at 92 percent memory by design. Your orchestrator restarts a container and the service does not blink.
Watch what happens meanwhile. The outage that matters arrives down a path no cause-alert anticipated, a config typo, an expired certificate, a dependency change, and nothing fires until your customers post about it.
Page on symptoms
Invert the logic and page on what your users experience. Error rate above a threshold. Latency percentiles breaching your objective. Checkout conversion falling off a cliff.
Notice the property that makes symptoms work. They are a small stable set that catches failures regardless of cause, including the causes nobody predicted.
Take the certificate expiry, the bad deploy and the database failover. All three announce themselves the same way, as errors and latency, so two or three symptom alerts cover the whole space of surprises.
Keep your cause-level signals in two supporting roles. Put them on dashboards, where they answer why after a symptom page has told you what.
Give the genuinely predictive ones a ticket, not a page. Disk 80 percent full and climbing means somebody should act this week, not that somebody must wake up now.
Route deliberately, because routing is the whole game. A page means a human must act now to protect users, and everything else is a ticket, a dashboard, or a deletion.
Audit your existing alerts with one question. When this last fired, did anybody do anything beyond acknowledging it? If the honest answer across the last ten firings is no, it is not an alert. It is a noise generator wearing an alert's clothes.
Worked example
Jonas inherits on-call for a document signing API with 34 alert rules, most of them thresholds on CPU, memory, and pod restarts. In his first week he is paged 11 times, and every single page is a false alarm, including three for a nightly PDF rendering job that always pins CPU for 20 minutes. Then on a Saturday the service actually breaks, a TLS certificate on an internal endpoint expires, and every signature request fails for 74 minutes before a customer email lands, because no rule watched error rate. The team rebuilds around two paging alerts, error ratio and p99 latency, the number the slowest one request in a hundred comes in under, against the SLO, and demotes the machine-level rules to a dashboard plus two disk-space tickets. Over the next quarter, pages drop from roughly 40 a month to 6, and every one of the 6 is real.