Five teams, all telling the truth
Here is the failure that created an entire industry.
A user reports that checkout takes four seconds. Checkout calls cart, pricing, inventory, payments and fraud, and several of those call further services of their own.
Ask each team and watch what happens. Each checks its own dashboard: percentiles healthy, error rate flat, nothing deployed. Five teams say not us, and every one of them is telling the truth as their tools see it.
Understand why your per-service metrics fail here. They aggregate across all callers.
The pricing service's slow percentile is computed over every request from every upstream, so the specific 800 millisecond calls that checkout makes with 40-item carts vanish inside that aggregate.
Your logs fail for a different reason. Without a shared identifier, correlating one request's lines across ten services means matching timestamps across machines whose clocks disagree slightly. At thousands of requests a second, five services logging within the same 50 milliseconds gives you dozens of plausible matches and no certainty.
Latency composes where nobody is looking
Look at the deeper problem, which is that latency composes in ways no single service can see. Sequential calls that could have run in parallel. One service calling another 200 times in a loop. A retry storm where a timeout fires three times before succeeding.
Notice where all three of those live: in the gaps between services, in the call graph itself, which is exactly what nobody's dashboard shows.
Want the request's own story instead. This request arrived at 14:32, spent 20 milliseconds in checkout, then 3,100 inside pricing's discount loop, then 60 in payments. Tracing is the machinery that records that story for real traffic, all the time, so your four-second mystery becomes a picture you scroll through.
Worked example
Lena is the tech lead when checkout p99 at her company creeps from 900 ms to 4 seconds over three weeks. She opens an incident channel and five service teams investigate for three days. Cart is healthy. Pricing is healthy. Payments points at fraud, fraud points back at payments, and everyone has a dashboard proving innocence. An engineer finally adds crude timing logs at checkout's call sites and finds pricing calls taking 3 seconds, but only for carts over 30 items. Pricing's own p99 never showed it because those calls are 0.4 percent of its traffic. Root cause: a discount rule shipped a month earlier queries a promotions table once per cart item, 30 items, 30 queries. Three days and five teams for a bug a single trace view would have shown as 30 stacked database spans in about ninety seconds.