You cannot keep everything
Tracing every request at scale is off the table.
A busy system generates tens of spans per request across billions of requests, and storing all of that costs more than the fleet you are observing. The paper that started this field made the case bluntly. They kept roughly one request in a thousand and still caught the problems that mattered, because high-volume issues turn up even in thin samples.
Decide at the front door, which is the simple approach. When a request arrives with no context, your edge flips a weighted coin and records the answer in the header. Every downstream service honours it, so the traces you keep are always complete.
Notice the painful blind spot in that. The decision happens before anybody knows how the request turns out.
Keep one percent and you also keep 99 percent fewer of your errors and slow requests, and those are exactly the traces incidents are made of.
Deciding at the end
Decide at the end instead and that flips. Every service sends every span to a collector tier, which buffers them until a trace completes and then looks at the assembled result. It keeps the interesting ones: anything with an error, anything over a latency threshold, plus a thin slice of normal traffic as a baseline.
Pay for it in infrastructure, and the cost is real. All the spans get generated and shipped regardless, and your collector holds state for every trace in flight. Traces must also be routed so one node sees every span belonging to a given identifier.
Start at the front door with a modest rate, and move to deciding at the end when your incident reviews keep landing on the interesting trace was not sampled. The tooling makes that switch a configuration project rather than a build.
Worked example
A media streaming company runs head-based sampling at 0.5 percent. During a March incident, playback start failures spike to 2 percent for 40 minutes, and the on-call finds a grand total of 6 sampled traces of failing requests, none from the worst-affected device type. The postmortem's action item: sampling must keep what incidents need. The platform team deploys the OpenTelemetry Collector with tail-based rules, keep every trace containing an error, everything over 2 seconds, and 0.1 percent of the rest. The collector tier costs about 3,000 dollars a month in compute and buffers spans for up to 30 seconds. Two months later a similar failure hits, and the on-call has 4,000 complete error traces to work with, showing a malformed manifest that only one smart TV firmware version choked on. Diagnosis takes 20 minutes.