When should something be a metric versus a log?

Metrics answer aggregate questions cheaply: rates, percentiles, saturation over time, things you alert on. Logs answer specific questions about individual events. The boundary is cardinality: if you want to slice by something with thousands of values, like user or merchant, that belongs in logs or traces, because each distinct label value in a metric becomes a stored time series.

Why does everyone say to alert on p99 instead of the average?

Because averages hide the tail where real pain concentrates. A handful of multi-second requests barely move an average over millions of fast ones, yet at scale that handful is thousands of affected users, biased toward your heaviest users because they make the most requests. Percentiles state directly what fraction of requests exceeded a threshold.

Pull-based scraping seems odd. Why not have services push metrics?

Pull gives you a built-in liveness check, since a failed scrape is an immediate, alertable signal that the target is gone, and it lets you run the same setup in dev by pointing Prometheus at an endpoint. Push fits short-lived batch jobs and heavily firewalled environments, which is why Prometheus offers a push gateway for exactly those cases.

Observabilitychapter 2 of 4 · 4 lessons

Metrics & Monitoring

Numeric time-series data to spot trends, alert on anomalies, and drive capacity planning.

← All chapters View in cheatsheet

Your dashboard says average response time is 99 milliseconds. It has said that all week, and it is true.

One customer in a hundred is waiting four seconds. At a million requests a day that is 10,000 people, and your heaviest users hit it most, because they make the most requests.

The average is not lying. It is just describing a number almost nobody experienced, and it will keep looking healthy right up to the day those customers leave.

Metrics are cheap numbers recorded over time, and they are the signal that tells you something is wrong before a customer does. Which numbers you record, and which shape you record them in, decides whether they can.

Lessons

4 in this chapter

The Four Golden SignalsLatency, traffic, errors, saturation: four curves that say whether users are hurting.2 min
Counters, Gauges, and HistogramsThree metric types cover almost everything, and one labeling mistake can take down your monitoring.3 min
Percentiles, Not AveragesAverage latency is a fiction; p99 is what your busiest users actually feel.2 min
Dashboards That Help at 3amA dashboard is an incident tool for a half-awake brain, not a wall decoration.2 min