Why 3 or 5 nodes and never 4?

Fault tolerance is floor((N-1)/2). Three tolerates 1 failure and four also tolerates only 1, because a majority of 4 is 3. The fourth node adds cost and write latency for no extra safety, and it makes an even split possible. Go 3, then 5, and only past that if you genuinely need to survive 3 simultaneous failures.

Is a Redis lock good enough for leader election?

Usually not on its own. A TTL lock guarantees at most one holder only if nobody pauses longer than the TTL, and garbage collection pauses break that assumption. Either use a system built for it, etcd or ZooKeeper, or add fencing tokens so the resource rejects stale writes rather than trusting the timing.

What actually happens when a cluster loses quorum?

It stops accepting writes. A 3-node cluster that loses 2 nodes has 1 survivor, which is not a majority, so it refuses rather than risk diverging from the nodes it cannot see. Reads may still be served depending on configuration. This is the availability you traded away for consistency, and it is why quorum members belong in separate failure domains.

Reliabilitychapter 5 of 5 · 4 lessons

Consensus & Leader Election

Getting a group of machines to agree on one value, so failover cannot produce two primaries.

← All chapters View in cheatsheet

Your database primary stops answering. Your health checker waits ten seconds, decides it is dead, and promotes a replica.

The primary was not dead. A network link dropped between it and the checker, and it has been happily taking writes from everybody on its side of that link the whole time.

You now have two machines that both believe they are the primary, both accepting writes to the same rows. Nothing has crashed and no error appears anywhere, because every component is doing exactly what you told it to.

When the link comes back you have two versions of the truth and no general way to merge them. Avoiding that afternoon is what consensus is for, and it starts by giving up on the idea that you can reliably tell a dead machine from an unreachable one.

Lessons

4 in this chapter

Split BrainA node that cannot be reached and a node that is dead look exactly the same.3 min
QuorumsRequire a majority, because two majorities cannot exist at the same time.2 min
How Raft Elects a LeaderTerms, randomised timeouts, and a majority of votes.3 min
Using It Without Building ItPut the decision in etcd, keep your data out of it.3 min