Counters, Gauges, and Histograms

Three shapes

Metrics come in three shapes, and you want to know all three cold.

A counter only goes up. Requests served, bytes sent, errors thrown. You never read its raw value, because four million requests since the process started means nothing.

Ask for its rate of change instead and you get requests per second. Counters survive gaps in collection gracefully, so the convention is to count events and derive rates at query time rather than exporting a rate you computed yourself.

A gauge moves both ways. Memory in use, queue depth, active connections, items in a cart.

Treat gauges as snapshots. They answer what is it now and what was it over time, and they need care around spiky values, because collecting every 15 seconds can miss a two second spike entirely.

A histogram exists because latency is a distribution and not a number. It keeps counters of how many observations fell into predefined buckets, under 10 milliseconds, under 50, under 100, and so on.

Get percentiles out of that across any time range and any set of machines. You choose the bucket boundaries up front and your accuracy depends on them, the price of keeping histograms cheap and mergeable.

The label that kills your monitoring

Avoid the trap in this whole system: label cardinality. Every unique combination of label values creates a separate series stored in memory.

Labels like region or status code multiply your series by five or ten, and that is fine. A label carrying a user identifier multiplies them by your user count, and your monitoring keeps an index entry for every one.

Understand why that failure is so vicious. Teams have run their monitoring out of memory by adding a single label, and monitoring dies first, right before you need it to debug the thing that killed it. Send high-cardinality questions to your logs and traces, never to metric labels.

// Counter: only goes up. Never read the raw value, ask for its rate.
requests.labels("checkout", "200").inc();
//   rate(http_requests_total[5m])  ->  requests per second

// Gauge: moves both ways. A snapshot, so it can miss a short spike.
queueDepth.set(queue.size());

// Histogram: latency is a distribution, not a number. Buckets are
// chosen up front, and percentiles are estimated from them later.
latency.observe(elapsed.toSeconds());

// The trap. Every unique label combination is a separate stored
// series, so this one line multiplies them by your user count and
// runs your monitoring out of memory, right before you need it
// to debug what killed it.
requests.labels(userId).inc();        // never do this

Worked example

A payments team at a marketplace adds a merchant_id label to their request counter so support can graph any merchant's traffic. In staging, with 50 test merchants, everything looks fine. Production has 380,000 merchants, and each one multiplies against the existing route, method, and status labels. Active series on the Prometheus server climb from 900,000 to 41 million over two days as merchants trickle through. Memory usage follows, and at 02:15 on Wednesday the server OOMs and restarts in a crash loop, taking dashboards and alerts down with it. The team is blind for 90 minutes, during an unrelated deploy no less. The fix: drop the label, cap the counter to coarse labels, and move per-merchant questions to the log pipeline, where cardinality is just another field.