Replication Lag in Practice

The gap is always there

Asynchronous copies are always a little behind, usually by milliseconds. Two things go wrong with that.

Sometimes the gap grows to minutes. And even a tiny gap breaks what your users expect, in specific ways you can reproduce.

The bug you will ship at least once

Meet the classic bug first, because you will ship it at least once. A user saves a change and the write lands on the leader. The confirmation page fires off a read, the balancer sends it to a copy that has not caught up, and the user watches their edit disappear.

So they do it again, and now you have two of everything, or a support ticket. Nothing is down. Every piece behaved exactly as configured. That is what makes lag bugs miserable: they come and go depending on which copy a request happened to hit, and they never show up while you are testing.

Two cheap fixes. Send that user's reads to the leader for a short window after they write, 5 or 10 seconds, remembered in their session.

Or have the write hand back the leader's log position, then make their later reads wait until the copy has caught up past it. Postgres and MySQL both let you do that. What does not work is assuming the gap is zero because it usually is.

The gap itself is worth watching, because it likes to spiral. A schema change, a bulk import, or one copy on tired hardware can take it from 50 milliseconds to 20 minutes, and every fix above quietly assumes it stays small.

It deserves a first-class alarm: your database reports how far behind each copy is, and a copy drifting should page you before your users find it. Remember one more thing for the next lesson. However far behind your copy is when the leader dies, that distance is your data loss.

// The bug: save, then immediately read from a replica that has
// not caught up, and the user watches their edit vanish.
void updateProfile(long userId, Profile p) {
  db.leader().save(p);
  session.set("wroteAt:" + userId, now());   // remember they just wrote
}

Profile read(long userId) {
  Instant wrote = session.get("wroteAt:" + userId);
  boolean recent = wrote != null && wrote.isAfter(now().minusSeconds(10));

  // Their own reads go to the leader for a few seconds. Everyone
  // else keeps the cheap path.
  return recent ? db.leader().find(userId) : db.replica().find(userId);
}

Worked example

Camille runs community forums where users edit posts. The bug reports read like ghost stories: "I fixed my typo, the page reloaded, the typo was back, then it was fixed an hour later." Every edit was actually working. The edit wrote to the leader, the post-save redirect read from a replica running about 2 seconds behind during peak, and users were shown the pre-edit version, so many re-edited or gave up. Reproduction took a week because staging had one database and no lag. The fix took an afternoon: after any write, the app sets a 10-second flag in the user's session, and the query router sends flagged users' reads to the leader. Ghost edits stop. Camille also adds a lag graph to the on-call dashboard, which later catches a dying replica disk two hours before users would have.