Redundancy & Failover
No single point of failure. A backup is always ready to take over.
Every disk in your rack has a percent or two chance of dying this year. You have forty of them, so something fails roughly every few months, and you have never noticed because it was always something with a spare.
Then the one machine without a spare goes.
Draw your architecture and circle every box that appears exactly once. The database. The load balancer. The certificate that expires on a Saturday. The one engineer who knows the production password.
Every circle is a bet that the thing never fails, and at a percent or two a year across enough components, you lose that bet on a schedule. Redundancy is buying spares. Failover is the much harder problem of deciding when to use one.
Lessons
4 in this chapter- Single Points of FailureAny component whose death takes the whole system down with it.2 min
- Active-Passive and Active-ActiveFrom a cold spare you boot in a panic to two live nodes sharing the load.2 min
- Health Checks and Split BrainFailover is only as good as its detector, and the detector can be fooled.3 min
- Failover DrillsAn untested failover path is a rumor, not a capability.2 min