Hot Keys

One key, more popular than one machine

A pop star posts, and four million people open the same profile in the same minute.

Look at your dashboards. Your cache is working perfectly and your hit rate is 99.9 percent, and one node is completely saturated while everything routed to it times out.

Meet the hot key, the opposite of every problem so far. Nothing is missing and nothing is stale. One entry is simply more popular than one machine can serve.

Understand why adding nodes does nothing, because this matters more than the fix. Sharding spreads different keys across machines by hashing each one, and a single key hashes to a single node by definition. Ten more servers give that key exactly the capacity it had before.

Making it more than one key

Make it more than one key, then. Write the same value under a handful of suffixed copies and have each reader pick one at random, spreading the reads across as many nodes as you made copies.

Pay for that on writes, which now have to update or delete every copy, so reserve it for the few keys that need it.

Or keep the value in each application server's own memory, the one place a hot key is an advantage. It is read constantly so it stays warm, and you have as many copies as you have servers.

Those copies disagree for the length of the expiry, which for a profile is usually fine and for a balance is not.

Detect it before it detects you. Redis has a scan for exactly this, and the shape in your own metrics is unmistakable: one node far busier than its peers while the cluster overall looks quiet.

// Write every copy. N is small: this is for the handful of keys
// that need it, not for the keyspace.
static final int COPIES = 10;

void putHot(String key, String value) {
  for (int i = 0; i < COPIES; i++) {
    redis.setex(key + ":" + i, 300, value);
  }
}

// Read one at random, so the load spreads across the nodes
// those copies happen to hash to.
String getHot(String key) {
  int pick = ThreadLocalRandom.current().nextInt(COPIES);
  return redis.get(key + ":" + pick);
}

// redis-cli --hotkeys finds them before an incident does.

Worked example

Arjun's version was a flash sale. One product went on the homepage banner at 10am and took 40 percent of all catalogue traffic for twenty minutes.

His Redis cluster had six nodes. Five of them sat under 20 percent busy while the one holding that product ran at 98 and started shedding connections, and the page failed for customers whose requests happened to need it. The cache never missed once.

He now writes banner products under ten suffixed keys during a sale and reads one at random. The same twenty minutes the following month spread across four nodes and peaked at 34 percent, and he only has to remember to do it for the one product on the banner.