Skip to main content
Observabilitychapter 1 of 4 · 4 lessons

Logging

Record discrete events to answer 'what happened and when' during incidents.

A customer emails: my payment failed at about half past two yesterday, order 8812.

You open your logs and search. There are 40 million lines from yesterday, and the one you want says payment failed for user after retry, with no order number, no error code and no way to tell which of the 900 payment failures that afternoon was theirs.

The line was written for a human reading one line. Nobody reads one line.

Logs are still the signal you reach for when you need to know exactly what happened to one specific request at 14:32 last Tuesday, and whether that is possible was decided long before the incident, by how the lines were written.

Lessons

4 in this chapter
  1. Structured LoggingLog JSON with named fields, not prose, so a machine can answer questions during an incident.3 min
  2. The Log PipelineFrom stdout to a search box: agents, buffers, and indexers, and where each one falls over.2 min
  3. Sampling and CostLog volume grows with traffic, the bill grows with volume, and sampling is how you stay solvent.2 min
  4. What to Log, and What Never to LogLevels with teeth, context that answers questions, and zero secrets or PII.3 min