Locks, Jitter, and Early Refresh

One recompute per expired key

Every defence enforces the same rule, one recompute per expired key, and they differ only in where they enforce it.

Enforce it at miss time with a lock. On a miss, a request tries to claim a per-key lock, and the winner queries the database and refills the cache while everybody else waits briefly and re-reads.

Do better than waiting: serve the just-expired value if you kept one. That is what CDNs ship as serving stale while revalidating, and it turns an expiry from a latency cliff into a non-event.

Give that lock a lifetime of its own, so a winner that crashes does not deadlock the key forever.

Jitter, and the keys you already fear

Attack mass expiry with jitter. Instead of a flat ten minutes, use ten minutes plus a random minute, and keys populated together now expire spread across a minute instead of a single second.

Ship it everywhere, because it is one line of code and there is rarely a reason not to.

Stagger recomputation per key with early expiry. As an entry ages, each read rolls a die whose odds rise as expiry approaches, and treats the entry as expired and refreshes it early.

Trust the statistics: one lucky request refreshes the key shortly before the deadline, and the herd never sees a miss. The published version weights those odds by how long the recompute takes, refreshing your expensive keys earliest.

Skip the cleverness for the handful of keys you already know are dangerous. Refresh them from a background job every few seconds, with a lifetime comfortably longer than the interval, and your request path never recomputes them at all.

Give scoreboards, home page payloads and config blobs that treatment. In practice you ship jitter everywhere, locking or early expiry on hot keys, and background refresh on the crown jewels.

Product get(long id) {
  String key = "product:" + id;
  Product hit = cache.get(key);
  if (hit != null) return hit;

  // Only one caller wins the lock and talks to the database.
  // NX = set only if absent. The expiry stops a crashed winner
  // from deadlocking the key forever.
  boolean mine = cache.setIfAbsent("lock:" + key, "1", Duration.ofSeconds(10));
  if (!mine) {
    Thread.sleep(50);
    return get(id);                    // the winner will have filled it
  }

  try {
    Product p = db.queryOne("SELECT * FROM products WHERE id = ?", id);
    // Jitter, so keys written together do not expire together.
    long ttl = 600 + ThreadLocalRandom.current().nextInt(60);
    cache.set(key, p, Duration.ofSeconds(ttl));
    return p;
  } finally {
    cache.delete("lock:" + key);
  }
}

Worked example

After the scoreboard outage, Chen's team applies defenses in layers. First, jitter: every SET gains a random 10 percent on its TTL, ending mass expiry from deploys. Second, the scoreboard and the five other hottest keys move to background refresh: a worker recomputes each every 3 seconds and writes with a 30-second TTL, so the request path never recomputes them; even if the worker dies, on-call has 27 seconds of served traffic to notice. Third, all remaining cache-aside misses go through a lock helper: SET NX with a 10-second lock TTL, losers serve the stale copy the helper stashes under key:stale. They replay the outage in a load test, expiring the scoreboard while the database is artificially slowed. Database queries for the key during the 4-second window: exactly 1, down from 160,000.