Can a system have low latency but low throughput at the same time?

Yes, and it is common. A single-threaded service might answer each request in 5 ms but only sustain 200 requests per second. Latency describes one request's speed; throughput describes total capacity. You raise the second with more workers or batching, which may not change the first at all.

Why do engineers obsess over p99 when it only affects 1 percent of requests?

Because exposure compounds. A user making dozens of requests per session will likely hit the tail, and a page that fans out to many backend services waits for the slowest one, so the tail gets sampled many times per page. At scale, 1 percent of requests is also a large absolute number of unhappy users.

Does adding more servers reduce latency?

Only if the latency was caused by queueing, meaning the system was running near its capacity. In that case more servers drain the queue and latency falls back to normal. If a single request is slow on an idle system, the slowness is in the request path itself, and more servers change nothing.

Fundamentalschapter 3 of 6 · 4 lessons

Latency vs Throughput

Why adding servers sometimes fixes a slow page and sometimes changes nothing.

← All chapters View in cheatsheet

You run a photo site. Somebody opens a profile, your server asks the database for that person's photos, and the page comes back. That round trip takes 200 milliseconds, and your four servers handle about 500 of those a second between them.

This quarter the profile page has been getting slower. It now takes 2 seconds. Traffic has not grown, no release went out, and support is filling up.

So the team doubles the servers, four to eight. The bill doubles. The page still takes 2 seconds.

Nothing was waiting for a free server, because there were always free servers. Every request was waiting on one database query that got slower as the photos table grew, and eight servers wait for that query exactly as long as four did. Buying the wrong one of these two things is the most expensive ordinary mistake in this field.

Lessons

4 in this chapter