Skip to main content
Latency vs Throughputlesson 4 of 4 · 3 min read

Trading One for the Other

Once latency and throughput are separate dials, you start noticing how many techniques turn one by deliberately turning the other the wrong way.

Batching

Batching is the one you will meet first. Writing a thousand database rows one statement at a time pays the per-statement overhead a thousand times. One bulk write pays it once and can move fifty times as many rows a second. The cost lands on the first row, which now waits for its 999 companions.

Message producers do the same trick, holding a batch open for a few milliseconds to fill it. More messages per second, and every individual message arrives a little later. For an analytics pipeline that trade is obviously right. For a fraud check holding up a checkout it is obviously wrong.

The real failure is making the trade without noticing, because that is the real failure. A client library that quietly buffers writes underneath a latency-sensitive endpoint has made the decision for you, and your metrics may not show it.

Parallelism and queues

Parallelism looks like a free win, and Amdahl's Law says how far it goes: your speedup is capped by whatever fraction of the work stays sequential. Parallelise 95 percent of a job perfectly and the remaining 5 percent still holds you to twenty times faster, however many machines you rent. Fanning out has its own tax, since the response waits for the slowest branch, which is the tail latency problem arriving again.

Queues are the third lever. Put one in front of a worker pool and you absorb bursts far above what you sustain, then smooth them out. That is a win for throughput and for staying up. Time spent in that queue is still latency, and it is invisible in your service's own metrics because the clock starts when work is picked up. Measure from the moment a job is enqueued, or your dashboards will insist everything is fine while people wait.

the shape of it
Event stream6k events/sBatch bufferflush 50 ms / 500Bulk INSERT500 rows/stmtPostgres1. append2. flush batch3. one round trip4. +50 ms latency
step 1 of 4
The buffer multiplies write throughput and charges every event up to 50 ms of added latency.

Worked example

Nadia owns an ingestion service at an ad-tech firm that writes click events to Postgres, its relational database, one row at a time: 2,000 rows per second at 3 ms each, and the service falls behind every evening peak of 6,000 per second. She adds a buffer that flushes every 50 ms or 500 rows, whichever comes first, using multi-row INSERTs. Throughput jumps to 30,000 rows per second on the same hardware, and the backlog problem disappears. Each event now waits up to 50 ms before hitting disk, which nobody downstream can even perceive for analytics data. Six months later a teammate reuses the same buffered writer for a payment audit log with a hard 100 ms end-to-end budget, and the 50 ms flush interval quietly eats half of it. Same code, same trade, opposite verdict.

Latency vs Throughput: wrapping up

In the real world

  • 01Kafka producers expose linger.ms and batch.size so operators can explicitly trade a few milliseconds of message latency for much higher publish throughput.
  • 02Amazon has published that additional page latency measurably reduced sales, which is why its internal services carry p99 latency SLOs rather than averages.
  • 03Google's Jeff Dean popularized the "numbers every programmer should know" latency table, and Google's tail-at-scale work describes hedged requests to cut p99 on fan-out reads.
  • 04Netflix pushes video bytes from Open Connect appliances inside ISP networks, turning a cross-country fetch into a nearby one, while its control plane stays centralized.
  • 05Redis serves reads from memory in well under a millisecond, and teams put it in front of relational databases precisely to move hot reads up the latency ladder.

Questions people ask

Can a system have low latency but low throughput at the same time?

Yes, and it is common. A single-threaded service might answer each request in 5 ms but only sustain 200 requests per second. Latency describes one request's speed; throughput describes total capacity. You raise the second with more workers or batching, which may not change the first at all.

Why do engineers obsess over p99 when it only affects 1 percent of requests?

Because exposure compounds. A user making dozens of requests per session will likely hit the tail, and a page that fans out to many backend services waits for the slowest one, so the tail gets sampled many times per page. At scale, 1 percent of requests is also a large absolute number of unhappy users.

Does adding more servers reduce latency?

Only if the latency was caused by queueing, meaning the system was running near its capacity. In that case more servers drain the queue and latency falls back to normal. If a single request is slow on an idle system, the slowness is in the request path itself, and more servers change nothing.

Quick review

Latency:
end-to-end time for a single request. p99 (worst 1%) matters more than averages for SLA
Throughput:
requests per second (RPS) the system can sustain
Latency hierarchy:
L1 cache ~1 ns → RAM ~100 ns → SSD random read ~100 μs → same-DC round trip ~500 μs → cross-region round trip ~100 ms
Batching improves throughput but hurts latency. Trade-off is intentional in Kafka, bulk DB inserts
Amdahl's Law:
overall speedup limited by sequential fraction. Parallelizing 95% gives max 20× gain
Little's Law:
Latency = Queue_Length / Throughput. Use to size thread pools and buffer sizes
Measure at p50, p95, p99, p999. Tail latency often determines user experience
the trade-off

High throughput often requires buffering/batching which adds latency. Pick one to optimize first.

in the room

Optimize latency for user-facing requests (< 200 ms target). Throughput for batch/async jobs.