What should page a human versus create a ticket?

Page only when a human must act now to protect users: SLO-threatening error rates, latency breaches, fast error budget burn. Everything predictive or slow-moving, disk filling over days, a slow budget leak, a flaky nightly job, becomes a ticket for business hours. If nobody would take action within the hour, it is not a page.

Do burn-rate alerts require having SLOs first?

Yes, and that ordering is the point. A burn rate is measured against an error budget, which comes from an SLO target over a window. If you have no SLO yet, pick a defensible starting number from a few weeks of real data, like 99.9 percent success over 30 days, and refine it. The act of choosing forces the availability conversation your alerting was avoiding.

How detailed should a runbook be?

Detailed enough that an engineer who has never touched the service can verify the problem and attempt mitigation without asking anyone: exact dashboard links, exact commands with expected output, and a named escalation path with a time limit. Skip long architecture essays. If a step needs deep context to execute safely, that step is really an escalation in disguise.

Observabilitychapter 4 of 4 · 4 lessons

Alerting

Notify on-call when SLO is at risk, before the customer notices.

← All chapters View in cheatsheet

At 03:14 the pager goes off. Processor usage on one machine crossed 80 percent. The on-call engineer acknowledges it from bed, half asleep, and goes back to sleep, because it is the fourth one this week and none of them meant anything.

At 04:02 it goes off again, and they acknowledge that one the same way. That one was checkout returning errors to every customer.

Nobody was negligent. Forty pages a week, almost all of them noise, trains a person to dismiss pages, and the reflex cannot tell the fortieth false alarm from the real thing.

An alert strategy is a filter: out of everything your telemetry could say, which few conditions are worth waking somebody for. Get it wrong one way and people stop trusting the pager. Get it wrong the other and nobody hears about the outage until morning.

Lessons

4 in this chapter

Symptoms, Not CausesPage on what users feel; graph the causes for the investigation afterward.3 min
Burn-Rate AlertsAlert on how fast you are spending error budget, so severity sets urgency automatically.3 min
Killing Alert FatigueEvery page that needs no action teaches the on-call to ignore the one that does.3 min
RunbooksThe difference between a 10-minute page and a 2-hour one is usually a document.3 min