Skip to main content
SLA / SLO / SLIlesson 3 of 3 · 3 min read

Choosing Good SLOs

Machines are the wrong thing to measure

Bad objectives measure machines. Processor above 80 percent, memory pressure, queue depth.

See what is wrong with all three. None of them is a promise to a user, and every one of them pages somebody for conditions users never notice.

Sit your objectives at the user boundary and describe outcomes. Checkout requests succeed. Search returns within 400 milliseconds. The uploaded photo is visible within 5 seconds.

Test each one by trying to finish the sentence the user experienced. If you cannot, it is an internal metric wearing an objective's badge.

Use percentiles for anything about latency, because averages are a hiding place. An average of 90 milliseconds coexists perfectly happily with a slowest percentile of 4 seconds, meaning one customer in a hundred has a miserable time on every single request.

Target the tail you actually care about, as a ratio: requests under 400 milliseconds, and set the ratio from what your traffic really does.

Where targets come from

Derive your targets from measurement rather than ambition. Measure where you are now, find where users actually complain, and set the target slightly tighter than the complaint line, usually near what you already achieve.

Avoid the aspirational number. Five nines on a system delivering 99.7 produces a permanently exhausted budget and a team trained to ignore it. Each added nine roughly multiplies your cost in redundancy, automation and on-call weight, so every one needs a business justification rather than a vanity one.

Watch the other direction too, because overperforming has its own cost. Users calibrate to what you deliver, not to what you promised.

Learn from the internal lock service that became so reliable that teams across the company built as though it could never fail. Its owners began deliberately taking it down to its stated objective, flushing out the hidden dependencies while the failures were still cheap.

Keep the portfolio small, two to four objectives per service. Twenty targets means none of them can drive a decision, and driving decisions, what gets built next, what wakes somebody up, when launches pause, is the entire point.

Worked example

Yuki inherits an on-call rotation averaging 40 pages a week for a five-person team, built from about 200 threshold alerts accumulated over six years: processor, disk, heap, queue depth, all of it. She runs a two-week audit and finds that 34 of the last 40 pages required no action at all, while an 11-minute checkout outage in the same period never paged anyone, because no alert watched checkout. The rewrite goes SLO-first: three targets, checkout availability at 99.95, search under 400 ms at p99, the number the slowest one request in a hundred comes in under, for 99 percent of requests, order webhook delivery within 60 seconds at 99.9. Paging moves to burn-rate alerts on those three; the 200 old alerts become dashboard panels. Pages drop to about 5 a week, and the next checkout regression pages in 4 minutes because now something is actually watching what users feel.

SLA / SLO / SLI: wrapping up

In the real world

  • 01Google SRE runs error budget policies where an exhausted budget freezes feature launches, and deliberately takes the Chubby lock service down to its SLO with planned outages so internal teams cannot build on the assumption it never fails.
  • 02AWS's EC2 SLA pays service credits on a sliding scale, 10 percent below 99.99 percent regional uptime, rising to 100 percent if uptime falls under 95 percent, a concrete example of an SLA as a contract with priced consequences.
  • 03The Google SRE workbook's multiwindow, multi-burn-rate alerting recommends paging at a 14.4x burn rate over one hour, the point where 2 percent of a 30-day error budget has gone in 60 minutes.
  • 04Cloudflare sells enterprise plans with a 100 percent uptime SLA, workable only because the remedy is service credits scaled to the outage rather than a literal promise that failure is impossible.
  • 05Slack commits to 99.99 percent uptime for its Business+ and Enterprise Grid plans and pays credits when it misses, with incident history published on its status page.

Questions people ask

Should my SLO be the same as my SLA?

No, the SLO should be stricter. The gap between them is your reaction time: you want internal alarms firing, and fixes underway, well before the contractual threshold where credits start accruing. A common pattern is an internal SLO one notch tighter than the external SLA, like 99.95 internal against 99.9 contractual.

We haven't missed our SLO in a year. Is that good?

It deserves investigation rather than celebration. Either the target is too loose to describe what users need, or you are overspending on reliability that could fund features, or dependents are quietly building on a level of service you never promised. Google's answer to the last problem was planned outages for Chubby, deliberately spending the excess so nobody could depend on it.

Why not just target 100 percent availability?

Because a 100 percent target means an error budget of zero, which forbids deploys, migrations, maintenance, and any risk at all, forever. Users also cannot tell 100 percent from 99.99 over a month; their own wifi fails more often than that. Every nine multiplies cost, so the right target is the cheapest one users genuinely cannot distinguish from perfect.

Quick review

SLI (Service Level Indicator):
what you measure. p99 latency, error rate, availability percentage
SLO (Service Level Objective):
internal target. 99.9% availability, p99 latency < 200ms
SLA (Service Level Agreement):
external commitment with consequences. Miss it → credits/penalties
Error budget:
SLO of 99.9% = 0.1% error budget = 8.7 hours/year downtime allowed
Burn rate alert:
if burning error budget 10× faster than expected, alert before SLO is breached
99% (2 nines):
87.6 hours/year downtime. 99.9%: 8.7 hours. 99.99%: 52 min. 99.999%: 5 min
Google SRE:
set SLO below what users expect. Spend error budget on reliability investment vs feature work
the trade-off

Each additional nine costs disproportionately more in infrastructure, oncall, and engineering effort.

in the room

Set SLOs before writing alerts. Derive alerts from SLO burn rate, not arbitrary thresholds.