The pricing compounds
Logging bills surprise people because the pricing compounds. You pay to ingest, you pay to index, you pay to retain, and your volume scales in step with your traffic.
Do the arithmetic at a few billion requests a month with chatty services and observability spend rivalling compute spend is not a horror story. It is the default outcome of never saying no.
Sampling with a rule
Sample, with a rule that protects the data you actually need. Errors are rare and precious, so keep every single one of them along with your warnings.
Routine success lines are statistically redundant, so keeping 1 to 5 percent of them preserves every trend while cutting most of your volume. Do not emit debug logging in production at all, though a per-service switch to turn it on for an hour during an investigation is worth building.
Sample per request rather than per line if you can. Decide at the edge whether this request is being logged, then keep or drop all of its lines together, so the requests you kept remain complete stories.
Decide at the end instead, better still. If the request errored or ran slow, keep everything. Otherwise keep one in a hundred. Most log routers do this with a little configuration.
Pull two more levers. Retention tiers often save you more than sampling does, since nearly every search targets the last two days.
And give each team an ingestion budget with a dashboard showing cost by service, which creates the feedback loop that stops one team's debug logging from taxing everybody else. One well-known company ran up a nine-figure observability commitment before clawing it back. That is what unbounded spend looks like at the top end.
Worked example
Rohan runs platform at a 40-person fintech startup. Their Datadog log bill hits 22,000 dollars a month, up from 6,000 six months earlier, and finance asks questions. Breaking ingestion down by service, one culprit stands out: the API gateway logs four INFO lines per request, and traffic tripled. Rohan ships three changes over two weeks. The gateway collapses its four lines into one canonical line per request, INFO is sampled at 2 percent at the Vector layer with errors kept at 100 percent, and indexed retention drops from 30 days to 15 with the rest archived to S3. The bill lands at 7,400 dollars the next month. During the quarter that follows, on-call engineers report zero investigations blocked by missing data, because every error still carries its full context.