Skip to main content
Disaster Recoverylesson 2 of 3 · 3 min read

Backups That Actually Restore

They fail silently

Backups fail silently and in bulk.

The scheduled job that has been erroring into an unread mailbox for three weeks. The dump that runs fine and captures the wrong database. The credentials that expired, the disk that filled, the retention policy that deleted the copy you needed.

See why every one of those stays invisible. Writing a backup gets exercised nightly. Reading one gets exercised never.

Learn from the canonical case, a well-documented 2017 incident where a company had five separate backup and replication mechanisms and, at the moment of truth, not one usable copy.

Replication is not backup

Separate two things people constantly conflate. Replication is not backup.

Your replica applies every write within seconds, including the dropped table, the ransomware encryption, and the bug that nulls a column. Replication protects you from hardware dying. Backups protect you from bad data, because only a backup lets you return to a moment before the mistake.

Keep both, and add point-in-time recovery, a base backup plus a continuous log archive, which shrinks the gap between your restore points from hours to minutes.

Place them carefully, the next trap. Three copies, two kinds of media, one somewhere else, a rule that survives because people keep relearning it.

Read somewhere else as a different failure domain, not a different rack. Backups in the same building burn with the primary. Backups in the same cloud account get deleted by the same compromised credentials, and that is why write-once storage and separate accounts exist.

Test the restore on a schedule, with a stopwatch. A drill answers questions no backup dashboard can: is the copy complete, does anybody know the procedure, and how long does it take?

Take that last number as your real recovery floor, and expect it to be a shock. Automate the drill if you can, restoring into a scratch environment weekly with row counts checked. A broken backup then pages you within days instead of announcing itself during the disaster.

the shape of it
Bad deletePrimaryReplicaapplies it tooBackupfrom before1. drops table2. copies the delete3. the only way back
step 1 of 3
A replica copies your mistake within a second; only a backup predates it.

Worked example

Sam's team backs up a 2.1 TB relational database with a nightly pg_dump to S3, and the dashboard has shown green for two years. Their first restore drill is humbling. Hour one: the restore host needs 3 TB of disk nobody provisioned. Hour two: pg_restore begins, and the single-threaded index rebuild forecast says 14 more hours. The drill finishes at 19 hours end to end, against a written RTO of 4. Nothing was broken, exactly; the backup was valid. It was just a format that could not be restored fast enough to matter. They switch to disk snapshots every 6 hours plus continuous WAL archiving, which restores in 70 minutes by attaching a volume and replaying the tail. The quarterly drill now has a standing calendar slot and a one-page runbook with last quarter's timing at the top.