The Problem a Cache Solves

Where the time actually goes

Those 400 milliseconds go somewhere specific. Your own code takes about 8 of them. The three database queries take roughly 30 between them. Everything else is queueing: requests sitting in line, waiting for a database that has run out of room to answer them.

That queueing is the part worth understanding. No single query got slower. Too many are arriving at once, and each one waits its turn.

The same rows, over and over

Now count what they ask for. Log one minute of traffic and you find 61,000 requests covering 214 different products. The busiest twenty products are two thirds of everything. Your database answered the question "what is product 8871" around four thousand times in that minute, and the answer never changed once.

That means something specific. You are not asking your database to do anything difficult. You are asking it to do something easy, over and over, and paying the full cost every time.

A faster database would help a little. Not asking again would help enormously.

// Nothing wrong with this. It just runs a million times a day,
// and four thousand of those runs ask for the same product.
Product getProduct(long id) {
  return db.queryOne(
    "SELECT id, name, price, stock FROM products WHERE id = ?", id);
}

Worked example

Arjun runs the catalogue service at an electronics retailer. Black Friday took him from 400 requests per second to 3,000. He had not deployed anything that week.

He pulled a minute of query logs and sorted by product. Twenty products accounted for 68 percent of the traffic, and the top one had been fetched 4,100 times. Every one of those 4,100 answers was identical, because nobody had edited that product since March.