Paths that rot quietly
Failover paths rot quietly.
The standby sized correctly last year is now two machine generations behind your primary. Your runbook mentions a host decommissioned in March. Your promotion script needs a credential that expired, or one that lives in the head of an engineer currently on a beach.
Notice why none of that surfaces. Normal operation never takes the path. It surfaces at 3am during the real incident, the most expensive possible moment to learn it.
Rehearsal
Rehearse, which is the entire cure. Schedule an exercise where you deliberately fail a component, in staging first and then in production, and run the actual recovery with the actual on-call people.
Measure two things: whether your automation works, and how long your humans take. Write down the timestamps. If your architecture claims a five-minute failover, the drill is what turns that claim into a number you can say out loud.
Go further with continuous failure if you want. One large streaming company has been killing random production machines during business hours since 2011. The theory is that a failure you meet weekly is a failure you have already engineered away.
Start small rather than copying their tooling. Kill one instance of one service while the owning team watches, with a written prediction of what should happen and a stop button in case it does not. Widen the blast radius only as your confidence grows.
Drill the organisational path too. Who declares the incident, who can approve evacuating a region, who talks to your customers? A failover that works technically and then stalls for 40 minutes waiting on a decision maker still blows your recovery target.
Worked example
Lena schedules her team's first game day: unplug the primary database in staging and follow the runbook, target 10 minutes. It takes 96. The promotion script fails immediately because it authenticates with a service account whose key was rotated in January. The one person who knows the new key location is on leave, so they page him anyway, which is its own lesson. Then the promoted replica buckles under write load because it was a cheaper instance type, provisioned back when it only served reads. Nothing in the 96 minutes was exotic, just drift accumulated over 14 months. The team fixes the credential store, upsizes the replica, and repeats the drill monthly. By the third run they hit 6 minutes, and when a real failover happens that autumn it takes 7.