The stack
The standard stack scrapes a metrics endpoint on each of your services every fifteen seconds or so, renders dashboards from that store, and routes alert rules to your pager.
Notice why pulling beats pushing here, which sounds backwards until you operate it. Your collector knows immediately when a target stops answering, which turns the service is silently gone into an alertable fact instead of an absence of data you notice too late.
Design for the person who was woken
Design your dashboard for its actual audience: an engineer woken by a page, running at maybe 40 percent, who did not write this service.
Structure it top down for that person. First row, is the service serving users, with error rate and latency percentiles against their targets. Second row, the four golden signals with deploy markers drawn over them, because a startling number of incidents line up with a deploy and the overlay answers that instantly.
Below that, saturation of every hard dependency: database, cache, queue, connection pools. Push the resource minutiae onto a separate drill-down page.
Follow a few rules that keep dashboards honest. Link every alert to the dashboard that puts it in context, so nobody hunts for graphs mid-incident.
Keep your colours and units consistent, seconds everywhere or milliseconds everywhere, because a brain at 3am will misread a mixed axis.
Curate a handful of dashboards instead of maintaining a graveyard of 200 personal ones. A stale dashboard does not merely fail to help. It actively misleads.
Put the on-call runbook link in the header while you are there.
Judge the result by the right test. Not how impressive it looks on the office television, but how many seconds it takes a stranger to answer whether this service is okay and, if not, what changed.
Worked example
After a messy incident where the on-call spent 25 minutes hunting through 14 dashboards, Farah's platform team at a travel booking site standardizes one layout for all 30 services: SLO row on top, golden signals with deploy markers second, dependency saturation third, links to runbook and drill-downs in the header. Grafana provisions the dashboards from a template in git, so every service gets the layout for free and drift is a code review problem. Three weeks later a 02:50 page hits a service the on-call has never touched. The top row shows errors breaching SLO, the deploy marker shows nothing shipped in 6 hours, and the dependency row shows the cache connections saturated. He restarts the leaking consumer per the linked runbook. Time from page to mitigation: 11 minutes, on a service he had never opened before.