Skip to main content
Consensus & Leader Electionlesson 1 of 4 · 3 min read

Split Brain

Nothing failed, and you have two primaries

Take a normal primary with two replicas and a checker that promotes a replica when the primary stops answering. Now put a network partition between the checker and the primary.

Watch what happens, because nothing here is broken. Your primary is fine, serving traffic to every client on its side of the partition. Your checker cannot see it, concludes it is dead, and promotes a replica.

You now have two primaries. Both accept writes. Neither is wrong from where it stands, and no error is logged anywhere, because nothing failed. The system is doing precisely what you told it to.

Look for the damage somewhere other than the duplication. It is in the reconciliation.

When the partition heals you hold two divergent histories over the same rows, with no general way to merge them. Taking the latest write silently discards whichever side lost, which for an inventory count or a ledger is data loss with a timestamp on it. Merging by hand means reading both logs and deciding what the truth was.

Why the obvious fixes do not work

State the root cause precisely, because it is what rules out the obvious fixes. Detecting failure over a network is impossible to do perfectly.

A missing heartbeat means the machine is dead, or the network dropped it, or the machine is in a long pause, and from outside those cases are indistinguishable. Longer timeouts do not fix that. They trade a fast wrong answer for a slow one.

Change the goal instead, and that is the shift consensus makes. You cannot detect failure reliably, so stop trying to know who is alive and start controlling who is permitted to act. At most one machine may ever behave as leader, whatever anybody believes.

the shape of it
App serversrack APrimary Aalive, isolatedPartitionswitch failureWatchdogsees silencePrimary Bjust promotedApp serverseverywhere else4,100 writesunreachablepromote2,700 writes
step 1 of 4
Both sides behave correctly in isolation, and the result is two write histories that cannot be merged.

Worked example

A payments team runs Postgres with a primary in rack A, a replica in rack B, and a watchdog that promotes on 30 seconds of silence. A top-of-rack switch fails and isolates rack A from the watchdog, but not from the application servers that also live in rack A. The watchdog promotes rack B at 14:31. For the next 19 minutes both databases take writes: rack A serves the app servers beside it, rack B serves everyone else. When the switch is replaced, 4,100 transactions exist on one side and 2,700 on the other, with overlapping ids. There is no merge, only a decision: they keep rack B, replay what they can from rack A's WAL by hand over two days, and issue 43 refunds for payments that were taken and then vanished. Nothing crashed. Every component did what it was designed to do.