Skip to main content
Disaster Recoverylesson 3 of 3 · 3 min read

DR Architectures by Budget

Four tiers

Four tiers organise this space, and the industry now talks in these terms whichever cloud you run on. Each tier keeps more of your system alive in the recovery region, so each recovers faster and costs more.

Start at the floor: backup and restore. Data copied to another region, and nothing else exists until disaster. Then you build the whole stack from scratch, from your infrastructure code if you are disciplined and from memory if you are not.

Expect hours to days to recover, data loss wherever your backup schedule puts it, and a standing cost of storage alone, meaning nearly nothing.

Move up to a pilot light, which keeps the hardest part alive. Your data replicates continuously into live databases in the second region, with the application infrastructure defined but scaled to nothing.

Scale up compute around that warm data on the day, and your recovery drops to tens of minutes, because loading the data, the slow step, is already done.

Run a warm standby if you can afford it: the full stack in the second region at maybe a tenth of capacity. Failover is a DNS change your provider's health checks can automate, plus scaling up, so you recover in minutes.

Notice the subtler thing you bought. That environment serves real health checks daily, so it rots far more slowly than a cold one ever could.

Active everywhere, and what it really costs

Go active in several regions at full capacity if the business genuinely needs it, splitting traffic across them. Losing a region then simply reweights routing, and your recovery numbers approach zero.

Count the honest cost there, and it is not the doubled infrastructure bill. It is the engineering: data across regions means replication lag, conflict handling, and a consistency decision on every write path. A handful of companies operate here. Most should not.

Pick per system, not per company. Payments on warm standby, the marketing site on backup and restore, and everything in between priced against the recovery time your business actually signed.

the shape of it
Backup + restoreRTO: hours-daysPilot lightRTO: tens of minsWarm standbyRTO: minutesActive-activeRTO: near zerodata kept livestack kept livefull capacity
step 1 of 3
Each tier keeps more of the system running in the second region, buying faster recovery at higher standing cost.

Worked example

Diego runs infrastructure for an e-commerce company doing about 80,000 dollars an hour at peak, and the board asks what happens if their primary cloud region goes down. Multi-region active-active prices out around 45,000 a month plus a re-architecture of the checkout data layer, which nobody wants to fund. He proposes pilot light instead: Aurora cross-region replicas for the two core databases, container images and infrastructure templates already present in the DR region, compute at zero. Standing cost lands near 6,000 a month. The first evacuation drill takes 51 minutes, mostly waiting on autoscaling and one hardcoded region string someone finds at minute 30. The second drill, after fixes, takes 22. The board gets a one-line answer: a region failure costs about 25 minutes of sales, and holding that risk costs 6,000 a month.

Disaster Recovery: wrapping up

In the real world

  • 01GitLab's January 2017 incident: an engineer deleted the wrong Postgres data directory, and all five backup and replication mechanisms turned out to be broken or misconfigured; they restored from a 6-hour-old snapshot that existed by luck and published the full postmortem.
  • 02The OVHcloud Strasbourg fire in March 2021 destroyed the SBG2 data center; customers whose backups lived in the same building lost the primary and the backup in the same hour.
  • 03Code Spaces shut down permanently in 2014 after an attacker with control of their AWS console deleted servers and the backups stored in that same account, a one-failure-domain lesson in the 3-2-1 rule.
  • 04Netflix runs active-active across multiple AWS regions and uses Chaos Kong drills that evacuate a region, keeping the failover path exercised rather than theoretical.
  • 05AWS's disaster recovery whitepaper defines the four standard strategies, backup and restore, pilot light, warm standby, and multi-site active-active, and pairs them with Route 53 health checks and failover routing for the DNS switch.

Questions people ask

I have database replicas. Do I still need backups?

Yes. A replica applies every write within seconds, including the accidental DROP TABLE, the ransomware encryption, and the application bug corrupting rows. Replication protects against hardware failure; only backups with point-in-time recovery let you rewind to before a mistake. Treat them as answers to different disasters, because they are.

How do I actually pick RTO and RPO values?

Price the outage with the business: revenue per hour of downtime, cost and legal exposure of losing an hour of data. Then price the tiers of protection and put the two numbers side by side. The right RTO and RPO are where those curves cross, and they will differ per system, which is why blanket company-wide targets usually mean overspending somewhere and gambling somewhere else.

Is multi-region active-active overkill?

For most companies, yes. Full region outages are rare, and active-active roughly doubles infrastructure cost while forcing hard consistency decisions onto every write path. Pilot light or warm standby captures most of the protection at a fraction of the cost and complexity. Go active-active when the revenue math genuinely demands near-zero RTO, not for the architecture diagram.

Quick review

RTO (Recovery Time Objective):
max acceptable downtime. How fast must you recover?
RPO (Recovery Point Objective):
max acceptable data loss. How much data can you lose (in time)?
Backup & Restore:
cheap. RTO hours to days. RPO hours. Just restore from S3 backups
Pilot Light:
minimal version of system always running in DR region. Scale up on failure. RTO minutes
Warm Standby:
scaled-down replica always running. Switch DNS on failure. RTO seconds
Multi-Site Active-Active:
full capacity in 2+ regions. Zero RTO, zero RPO. Most expensive
AWS:
Route53 health checks + failover routing policy for DNS-based failover. Use S3 cross-region replication for data
the trade-off

Active-Active multi-region is expensive (2× infrastructure) and complex (distributed consistency). Match cost to actual requirement.

in the room

Define RTO/RPO in your SLA first, then pick DR strategy. Each additional '9' of availability costs significantly more.