Not a bell curve
Latency does not follow a bell curve.
Real request latency has a long right tail. Most requests finish fast, and a small fraction take ten or a hundred times longer, thanks to garbage collection pauses, cold caches, lock contention and slow disks.
Average that shape and you produce a number almost no request experienced. Ninety-nine requests at 50 milliseconds and one at 5 seconds averages to 99 milliseconds, which describes neither the happy majority nor the suffering minority.
Use percentiles, because they describe the distribution honestly. The median is your typical experience, and the higher percentiles are the tail, and the tail is where your reputation lives.
Count the people in it. At a million requests a day, pain at the 99th percentile hits 10,000 requests. Worse, your heaviest users make the most requests, so the chance your most valuable customers hit the tail in any session is far higher than one in a hundred.
Fan-out amplifies it
Watch fan-out amplify that brutally. If rendering one page calls 100 backends in parallel and each is slow one percent of the time, your page waits on the slowest, and the odds at least one lands in the tail are about 63 percent.
Read what that means. Your backend's 99th percentile just became your user-facing median. The countermeasures include firing a duplicate call when the first one is slow.
Avoid one operational trap: you cannot average percentiles. The mean of ten machines' 99th percentiles is statistically meaningless.
Merge the underlying histogram buckets across your machines first, then compute the percentile from the combined distribution, which is exactly what your metrics backend's quantile function does.
Worked example
A dashboard team at a logistics company reports average API latency of 120 ms and considers the service healthy. Meanwhile, their three biggest customers keep complaining that the dashboard feels slow, and sales forwards the complaints weekly. Tomas finally graphs percentiles instead: p50 is 85 ms, but p99 is 4.1 seconds. The page fans out to 40 widget queries in parallel, so nearly every page load for a data-heavy customer waits on at least one tail query, and big customers have the most data per widget. The average was diluted by millions of tiny requests from small accounts. The team caches the three heaviest widget queries and adds a 500 ms timeout with a partial-render fallback. p99 drops to 600 ms, and the complaints stop within two weeks. The average barely moves.