Database Replicationlesson 4 of 4 · 3 min read

Failover and Multi-Leader

The fire drill

Failover is the fire drill this whole chapter exists for. The leader dies, a follower gets promoted, clients point at the new one. Tools automate the steps, and every step hides a trap.

Detection is guesswork. A leader that has missed its heartbeat for 10 seconds might be dead, might be busy, might be fine but briefly unreachable.

Call it dead too eagerly and you promote a new leader while the old one is still happily accepting writes. Now two machines take writes for the same data, their histories drift apart, and a human unpicks the mess by hand. Making certain the old leader cannot accept another write before you promote anyone is the unglamorous detail that separates real failover from a demo.

Picking the replacement costs you something. You promote whichever follower is furthest along, and with asynchronous copies even that one is missing the leader's last few writes.

Failover turns your replication gap directly into lost data, which is the strongest practical argument for keeping a synchronous copy of anything you truly cannot lose.

Multi-leader, and when it earns its place

Consider multi-leader only when you have to. It takes writes on several machines, usually one leader per region, so writes are fast everywhere and losing a whole region gets simpler to survive.

You have also brought back the exact problem one leader was avoiding. Two leaders can accept conflicting changes to the same row, and nobody finds out until their logs meet. Resolving by last writer wins silently throws one of them away. Merging properly in your application is principled and expensive to build.

Be honest about when it earns its place. Multi-leader is for a genuine need to write in several regions, not for ambition. If your users can live with writes crossing an ocean, take one leader plus regional read copies. That is the design that will not page you at 4 am to referee two versions of a row.

the shape of it

step 1 of 2

Two leaders can accept conflicting writes, and nobody finds out until the logs meet.

Worked example

GitHub's October 2018 incident is the canonical failover story. A 43-second network partition cut GitHub's US East Coast data center off from the others. Orchestrator, their MySQL failover tool, did its job: it promoted replicas on the West Coast and redirected writes there. But during those 43 seconds the East Coast primaries had accepted writes that never replicated out, while the newly promoted West Coast leaders were already accepting new writes of their own. Two divergent histories, roughly 950 writes apart. Rather than discard either side, GitHub degraded the site for over 24 hours while engineers restored from backups and reconciled the divergence by hand. The public postmortem is worth reading whole: correct tools, correct promotions, and a topology where 43 bad seconds cost a day.

Database Replication: wrapping up

In the real world

01GitLab's 2017 database outage began as replication lag troubleshooting and ended with an accidental deletion on the primary; of five backup mechanisms none worked cleanly, about six hours of data was lost, and the full postmortem is public.
02Amazon RDS Multi-AZ is synchronous replication productized: the standby acknowledges before commit, and failover repoints DNS in one to two minutes without losing acknowledged writes.
03Facebook runs MySQL with a single writable primary per shard and replica fleets across regions, serving reads locally through TAO while writes forward to the primary's region.
04Kafka exposes the same trade through producer acks: acks=1 gives async-style speed with loss risk on leader failure, while acks=all waits for the in-sync replica set.
05Postgres ships the dial per transaction as synchronous_commit, so one database can give the payments ledger synchronous safety while the event log keeps async speed.

Questions people ask

How much replication lag is normal?

Milliseconds to low tens of milliseconds for replicas in the same data center under normal load. It spikes during bulk writes, schema migrations, and replica hardware trouble, sometimes to minutes. Treat lag as a monitored metric with an alert, because every read-from-replica strategy quietly assumes it stays small.

Does replication mean I don't need backups?

No. Replicas replay everything, including your mistakes, so a bad DELETE or dropped table propagates to every copy within seconds. Backups plus point-in-time recovery protect against human error and logical corruption; replication protects against hardware failure. GitLab's 2017 incident is the standard cautionary tale for confusing the two.

When is multi-leader replication actually worth it?

When you genuinely need low-latency writes in multiple regions, or clients that write while offline and sync later, and you are prepared to own conflict resolution. If write latency to one region is tolerable, a single leader with regional read replicas gives you most of the benefit with none of the conflict machinery.

Quick review

Single-leader (Primary-Replica):: all writes to primary, reads from replicas. Simple, eventual consistency on replicas
Replication lag:: async replicas may be seconds behind. 'Read your own writes' problem if reading from replica immediately after write
Synchronous replication:: primary waits for follower ack before confirming write. Zero lag but adds latency
Asynchronous replication:: primary doesn't wait. Lower latency, risk of data loss if primary crashes before sync
Multi-leader:: multiple primaries in different regions. Each accepts writes. Conflict resolution required (last-write-wins, CRDT)
Leaderless (Dynamo-style):: any node accepts writes. Quorum: W + R > N for consistency (e.g., W=2, R=2, N=3)
Replica uses:: read scaling, failover, analytics on replica (no read impact on primary), geo-local reads

the trade-off

Async replication = potential stale reads or data loss. Sync replication = latency increase.

in the room

Read-heavy workloads (80/20 read/write). High availability. Disaster recovery.

the short version

Promoting a follower is easy. Fencing the old leader and merging conflicts is the hard part.