Agree what the levels mean
Log levels only help if your team agrees what they mean.
Write the contract down. Error means a human should eventually look at this. Warning means something unexpected happened and the system coped. Info records the normal milestones of handling a request. Debug is for your laptop and for short production investigations.
Watch the level teams always break: error. If your services emit thousands of them on a healthy day, the level carries no signal at all, and the one line that mattered scrolls past unread during the outage.
Demote or delete any error nobody would ever act on.
Log at decision points and boundaries, because content matters more than volume. Request received and completed. External call made, with its duration and outcome. Retry attempted. Fallback taken. Background job started and finished.
Include the identifiers somebody will search by, because a line missing its identifiers answers no questions.
The never list
Keep the never list short and non-negotiable. Passwords, session tokens, API keys, authorization headers, full card numbers, government identifiers, and personal data like emails and addresses.
Treat your logs as the leakiest store you run. They get copied into tickets, pasted into chat, shipped to third-party vendors, and retained for a year, all with far broader access than your production database.
Personal data in logs is regulated data under privacy law, which makes your retention policy a compliance question. Log opaque identifiers rather than emails, and hash anything you need for correlation but not for reading.
Build defence in depth instead of trusting people. Redaction filters in your logging library for known field names, scrubbing rules in the pipeline agent, and a scanner that alerts when something token-shaped turns up in the index.
Take the cautionary tale seriously. One large company disclosed in 2019 that hundreds of millions of user passwords had been sitting in plain text in internal logs, searchable by around 20,000 employees. That failure is always one lazy debug line away.
Worked example
During a routine security review, Priya greps the log index of her employer's B2B API platform for the string "Bearer " and gets 1.4 million hits. A debugging middleware added eight months earlier logs full request headers, Authorization included, on every 4xx response. The blast radius calculation is grim: those logs flow to a third-party vendor and are retained for 90 days, so roughly 12,000 distinct customer API tokens are sitting in an external system. The company rotates every affected token, forcing 400 customers through a key rotation with an awkward email explaining why. The engineering fix takes a day: a denylist in the logging library that redacts Authorization, cookie, and password fields, plus a weekly scanner that alerts if anything matching a token pattern lands in the index again.