Skip to main content
Latency vs Throughputlesson 1 of 4 · 3 min read

Two Different Questions

Stay with the photo site, because both numbers describe it and they describe different things.

Latency: how long one request takes

Latency is how long one request takes, start to finish. One person opens one profile and waits 200 milliseconds. That is the latency, and it is measured in units of time.

Throughput: how many finish each second

Throughput is how many requests finish each second across the whole system. Your four servers get through 500 profiles a second. That is the throughput, and it is measured in requests per unit of time.

They answer different questions. Latency answers how long am I waiting. Throughput answers how many of us can be served at once.

Why the extra servers changed nothing

For the team who doubled their servers, that difference was the whole bill. Doubling servers raises how many requests finish each second, from 500 to about 1,000. It does nothing to the 2 seconds any single request spends waiting on a slow query, because that query takes 2 seconds on all eight machines just as it did on four.

They bought throughput. They had a latency problem.

The motorway

A motorway makes the same point if the servers are not sticking. Latency is your own drive time from one junction to the next. Throughput is how many cars pass a point each hour.

Add a lane and more cars get through every hour, while your own drive time on an empty road does not change at all. Raise the speed limit and your drive time falls, while the cars per hour barely moves. Different levers, different outcomes, same road.

Where the two meet: queueing

Watch where the two do meet, because they meet at queueing. Push your site past the 500 a second it can actually finish and requests start waiting in line before anybody even looks at them.

Follow the arithmetic, because it is unforgiving. At 600 requests a second arriving into a system that finishes 500, the line grows every second, and the wait to get into a server becomes most of what your user experiences. Latency climbs, gently at first, then almost vertically.

The causation runs the way people do not expect. A capacity problem, meaning not enough throughput, shows up on your dashboard as a latency problem, meaning everything feels slow.

Little's Law ties the two together: the number of requests in flight equals throughput times latency. Your site at 500 a second and 200 milliseconds each has 100 requests in flight at any moment, which is how many workers or connections you actually need.

The question to ask first

So ask one question first when somebody says the site is slow. Is it slow for one person at 2am, or slow only at the evening peak?

Slow at 2am, when nothing is queueing, is a latency problem inside the request. Slow only at peak is usually a capacity problem wearing latency as a costume, and that is the one where more servers genuinely helps.

the shape of it
Readers500 profiles/s4 serversnever all busy8 serversafter doublingPhotos tableone query: 2 sopen profilewait 2 swait 2 sstill 2 s
step 1 of 4
Doubling the servers doubled how many requests can wait at once. Each one still waits 2 seconds for the same query.

Worked example

Tomás runs the checkout service at a mid-size retailer. It handles 300 requests per second at a comfortable 80 ms each. A flash sale pushes traffic to 900 per second, the service saturates at around 500, and queues form; latency hits 4 seconds and carts get abandoned. His teammate proposes optimizing the payment call from 60 ms to 30 ms. That would be lovely on a quiet day and nearly useless right now, because each server would still saturate, just at a slightly higher ceiling. Tomás instead scales from 4 pods to 12, pushing sustainable capacity past 1,200 per second. Latency falls back to 80 ms untouched, because it was never the code that got slower, only the line in front of it.