Skip to main content
Logginglesson 4 of 4 · 3 min read

What to Log, and What Never to Log

Agree what the levels mean

Log levels only help if your team agrees what they mean.

Write the contract down. Error means a human should eventually look at this. Warning means something unexpected happened and the system coped. Info records the normal milestones of handling a request. Debug is for your laptop and for short production investigations.

Watch the level teams always break: error. If your services emit thousands of them on a healthy day, the level carries no signal at all, and the one line that mattered scrolls past unread during the outage.

Demote or delete any error nobody would ever act on.

Log at decision points and boundaries, because content matters more than volume. Request received and completed. External call made, with its duration and outcome. Retry attempted. Fallback taken. Background job started and finished.

Include the identifiers somebody will search by, because a line missing its identifiers answers no questions.

The never list

Keep the never list short and non-negotiable. Passwords, session tokens, API keys, authorization headers, full card numbers, government identifiers, and personal data like emails and addresses.

Treat your logs as the leakiest store you run. They get copied into tickets, pasted into chat, shipped to third-party vendors, and retained for a year, all with far broader access than your production database.

Personal data in logs is regulated data under privacy law, which makes your retention policy a compliance question. Log opaque identifiers rather than emails, and hash anything you need for correlation but not for reading.

Build defence in depth instead of trusting people. Redaction filters in your logging library for known field names, scrubbing rules in the pipeline agent, and a scanner that alerts when something token-shaped turns up in the index.

Take the cautionary tale seriously. One large company disclosed in 2019 that hundreds of millions of user passwords had been sitting in plain text in internal logs, searchable by around 20,000 employees. That failure is always one lazy debug line away.

the shape of it
Log lineLibrary filterknown field namesPipeline scrubpattern rulesIndexScanneralerts on tokens1. first pass2. second pass3. stored4. watched anyway
step 1 of 4
Three layers, because one lazy debug line is all it takes to leak a credential.

Worked example

During a routine security review, Priya greps the log index of her employer's B2B API platform for the string "Bearer " and gets 1.4 million hits. A debugging middleware added eight months earlier logs full request headers, Authorization included, on every 4xx response. The blast radius calculation is grim: those logs flow to a third-party vendor and are retained for 90 days, so roughly 12,000 distinct customer API tokens are sitting in an external system. The company rotates every affected token, forcing 400 customers through a key rotation with an awkward email explaining why. The engineering fix takes a day: a denylist in the logging library that redacts Authorization, cookie, and password fields, plus a weekly scanner that alerts if anything matching a token pattern lands in the index again.

Logging: wrapping up

In the real world

  • 01Stripe emits one canonical log line per API request carrying dozens of fields, and their engineers query those wide events instead of stitching together scattered INFO lines.
  • 02Datadog's 2023 earnings call disclosed a cryptocurrency customer, widely reported to be Coinbase, that had committed around 65 million dollars a year before optimizing its usage down.
  • 03Facebook disclosed in 2019 that 200 to 600 million user passwords had been stored in plaintext in internal logs, accessible to roughly 20,000 employees, triggering a multi-year cleanup.
  • 04Grafana built Loki to index only labels rather than full log content, trading slower queries for storage costs roughly an order of magnitude below Elasticsearch.
  • 05Cloudflare ingests logs for millions of HTTP requests per second into ClickHouse, using aggressive sampling and columnar storage to keep analytics queryable at that volume.

Questions people ask

Is structured logging worth it for a small app?

Yes, because the cost is nearly zero and retrofitting is painful. Every mainstream language has a JSON logging library, so structure costs you a dependency and a habit. The payoff arrives the first time you need to count errors by type or follow one request across two services, which happens well before you consider yourself at scale.

Won't sampling make me miss the one log line that explains an incident?

Not if you sample by outcome. Keep 100 percent of errors and slow requests and sample only routine successes, which are statistically interchangeable. The riskier gap is dropping lines under load because your pipeline has no buffer, which loses data exactly when you need it and without any policy behind the loss.

How long should I keep logs?

Match tiers to how logs are used. Almost all searches hit the last day or two, so keep 7 to 14 days hot, a month or three warm and cheaper, and archive to object storage for whatever your compliance rules demand, often a year. Keeping everything hot forever is the most common and least examined observability expense.

Quick review

Structured logs (JSON) > plain text. Queryable by field in Kibana/Splunk without regex hacks
Log levels:
DEBUG (local dev only), INFO (normal operation), WARN (unexpected but recoverable), ERROR (requires investigation), FATAL (process crash)
Correlation ID:
attach request_id to every log line in a request's lifetime. Pass via X-Request-Id header between services
Centralize with ELK stack (Elasticsearch + Logstash + Kibana) or Loki + Grafana (lower cost, label-based)
Never log PII (emails, passwords, SSNs, tokens). Violates GDPR/CCPA. Use pseudonymization or hash before logging
Sampling:
log 100% of ERRORs, sample 1% of DEBUG/INFO in production to control costs
Log retention:
hot storage 7 to 14 days, warm 30 to 90 days, cold archive 1 year. Match to compliance requirements
the trade-off

Logging is expensive at scale. Datadog/Splunk costs can exceed infrastructure costs. Sample aggressively.

in the room

Post-incident debugging. Audit trails for compliance. Understanding user behavior patterns.