Skip to main content
Latency Referencelesson 3 of 3 · 3 min read

Spending Latency Numbers in an Interview

Nobody quizzes the table

Nobody quizzes the table for its own sake. They want to watch you justify a design decision with a number instead of a habit.

The move is always the same. State your latency budget, add up what the design spends, and let the arithmetic choose the architecture.

The budget comes first. A user-facing request usually gets 200 to 500 milliseconds end to end before the product feels slow, and your backend's slice is maybe half of that once you subtract the encrypted handshake, the browser's rendering and the last mile. So you are allocating something like 100 to 200 milliseconds across every hop.

Spending the budget

Now spend it, and watch for the trap, which is that sequential calls add up. Five calls in a row inside one data centre at roughly a millisecond each cost about 5 milliseconds, which is nothing. Five calls in a row across regions at 100 milliseconds each cost 500, and your budget is gone before the database is involved.

That single contrast justifies three standard patterns in one breath. Put chatty services near each other. Make independent calls at the same time rather than one after another. Never put a cross-region hop inside a synchronous loop.

A cache is justified the same way, with a number. A database read that touches disk costs around 100 microseconds of storage time plus the query work, commonly a few milliseconds in total. A cache in the same data centre answers in about half a millisecond, and one inside your own process in nanoseconds. If 90 in every 100 reads hit that cache, your database sees a tenth of the traffic.

Any value you forget can be rebuilt from the bands: memory in nanoseconds, disk and the local network in microseconds, spinning disks and long-haul links in milliseconds. Getting the exponent right earns you more credit than reciting digits.

Worked example

Nadia is asked to design a global checkout flow in an interview. She states a budget out loud: 300 ms end to end, 150 ms for the backend. Her first sketch has the US checkout service synchronously calling an inventory service in Frankfurt, and she catches it with her own numbers: 100 ms cross-Atlantic round trip, twice (reserve, then confirm), is 200 ms, already over budget. She restructures: inventory replicates to each region asynchronously, checkout reads the local replica in about 1 ms, and the rare oversell gets handled by a compensation flow. The interviewer pushes back, asking why not just accept the latency. Nadia answers with the ladder: the original design spends two thirds of its budget on two trips across the network, each of which a nearby copy makes nearly free. The feedback afterward specifically calls out the budget arithmetic.

Latency Reference: wrapping up

In the real world

  • 01The canonical table comes from Jeff Dean's 'Latency Numbers Every Programmer Should Know,' circulated at Google to make engineers justify designs with orders of magnitude.
  • 02Google's published 'Speed Matters' experiments found that slowing search results by 400 ms measurably reduced how many searches users ran, which is why latency budgets are product decisions, not just infrastructure ones.
  • 03Amazon engineers famously reported that every 100 ms of added page latency correlated with about 1 percent of lost sales, a number still quoted in latency budget discussions.
  • 04CDNs like Cloudflare and Akamai exist to move content down the ladder: they convert a 100 to 200 ms cross-region fetch into an edge hit measured in tens of milliseconds or less.
  • 05AWS builds availability zones a few milliseconds apart within a region precisely so synchronous replication stays in the low millisecond band instead of the cross-region 100 ms band.

Questions people ask

Do I need to memorize the full latency table for interviews?

No. Memorize the bands (memory is nanoseconds, SSD and local network are microseconds, disks and cross-region are milliseconds) plus a few anchors: RAM 100 ns, SSD read 100 microseconds, same-datacenter round trip 500 microseconds, cross-region 100 to 200 ms. Everything else can be reconstructed from ratios, and interviewers care about the exponent, not the digits.

Why can't cross-region latency be engineered away?

The floor is physics. Light in fiber travels about 200 km per millisecond, and real routes are longer than great-circle distance. New York to Amsterdam is nearly 6,000 km of path, so roughly 60 ms round trip is the theoretical best, and real networks add routing and queuing on top. The only fixes are moving data closer to users or taking the hop off the synchronous path.

Are these numbers still accurate on modern hardware?

The absolute values drift; NVMe drives beat 100 microseconds and fast networks beat 10 microseconds per KB. The ratios are far more stable: memory stays about 1000x faster than durable storage, and anything leaving the datacenter stays in a different band entirely. Design decisions rest on the ratios, so the table keeps working even as the digits age.

Quick review

L1 cache hit:
~1 ns | L2 cache: ~4 ns | L3 cache: ~40 ns
Main memory (RAM) access:
~100 ns
SSD random read:
~100 μs | HDD seek: ~10 ms (100× slower than SSD)
Mutex lock/unlock:
~25 ns | Compress 1 KB with Snappy: ~3 μs
Send 1 KB over 1 Gbps network:
~10 μs | Same-datacenter round trip: ~500 μs
Cross-region:
US → EU ~100 ms | US → Asia ~200 ms | World round-trip ~500 ms
Key insight:
RAM is 1000× faster than SSD, and a same-datacenter round trip is 5000× slower than RAM. Cache decisions follow from this
Time units:
1 ms = 1,000 μs = 1,000,000 ns = 1,000,000,000 ps
in the room

Justify every design choice with latency math. 'Why cache?' → RAM 100ns vs DB disk 100μs = 1000× speedup.