Reliabilitychapter 3 of 5 · 3 lessons
Disaster Recovery
Plan for full region failure. RTO and RPO define how fast and how much data you can afford to lose.
At 11:20 an engineer runs a cleanup script against the wrong environment and drops the production database.
Your replicas do not save you. They applied that delete faithfully within a second, all three of them, because that is their entire job.
What saves you is a backup taken before 11:20, and the answer to two questions nobody has asked yet: how long until people can use the site again, and how much of this morning's data are you about to lose forever.
Both are numbers, both belong to your business rather than to you, and the worst time to discover them is now.
Lessons
3 in this chapter- RTO and RPOTwo numbers the business picks and engineering prices: how long down, how much data gone.3 min
- Backups That Actually RestoreNobody wants backups; everybody wants restores. Only one of those gets tested by default.3 min
- DR Architectures by BudgetFour standard tiers, each buying a better RTO with a bigger bill.3 min