Backups Do Not Prove Recoverability

Aug 25

By

database engineeringreliabilitydisaster recoverytechnical leadership

A green “backup completed” metric proves that bytes were written somewhere. It does not prove that the bytes are complete, decryptable, compatible with today’s application, or recoverable inside the business deadline.

The production invariant is stronger:

After a declared class of failure, we can restore an internally consistent service to an approved point in time, within an observed duration, using people and systems available during the incident.

That is a recovery capability. A backup is one input.

Start with loss, not tooling

Two numbers frame disaster recovery:

  • Recovery point objective (RPO): the maximum acceptable data loss measured in time.
  • Recovery time objective (RTO): the maximum acceptable time until the capability is usable again.

Do not assign one pair to an entire company. Authentication, checkout, analytics, and a documentation site rarely have equal business impact.

CapabilityRPORTOReason
payment ledgernear zero30 minmoney and reconciliation
customer workspace5 min2 hcore product continuity
analytics24 h24 hrecomputable output

These are business decisions expressed as engineering constraints. If leadership cannot explain the cost of losing four hours of data, engineering cannot rationally price a four-hour RPO.

Model the whole recovery graph

Restoring PostgreSQL is not the same as restoring the product. A usable recovery may require:

identity and secrets
        ↓
network and compute
        ↓
database + object storage + event log
        ↓
schema-compatible application
        ↓
workers, search indexes, caches
        ↓
DNS, traffic, reconciliation

Every edge is an ordering constraint. If the database is restored but encryption keys were deleted with the primary account, the backup is inert. If the data is restored to 10:05 but Kafka consumers replay effects from 09:40, the service can duplicate external actions.

Maintain a machine-readable recovery manifest with:

  • source and target environment;
  • backup identifier and checksum;
  • application and schema version;
  • encryption-key dependency;
  • recovery target time;
  • replay boundaries for brokers and workers;
  • validation queries;
  • traffic-switch owner.

The manifest turns tribal knowledge into an executable contract.

PostgreSQL PITR needs an unbroken history

PostgreSQL point-in-time recovery combines a base backup with archived write-ahead log (WAL). PostgreSQL’s documentation is explicit: recovery depends on the required continuous sequence of WAL files. A valid base backup plus a missing WAL segment cannot satisfy the intended recovery point.

Monitor the recovery chain, not just the latest object:

SELECT
  now() - last_archived_time AS archive_age,
  archived_count,
  failed_count,
  last_failed_wal
FROM pg_stat_archiver;

Alert on archive age relative to RPO. Validate object checksums and retention. Keep recovery credentials separate from the failure domain of the production control plane.

A replica is not a backup. Replication quickly copies valid writes, accidental deletes, and some forms of corruption. It improves availability; it does not create historical recovery points by itself.

Restore into isolation first

Never make the first restore attempt directly into the production destination. Restore into a quarantined environment where automation cannot emit email, charge cards, call webhooks, or consume live queues.

Validation should test business invariants, not merely database startup:

-- Ledger must balance per currency.
SELECT currency, sum(debit_cents) - sum(credit_cents) AS imbalance
FROM ledger_entries
GROUP BY currency
HAVING sum(debit_cents) <> sum(credit_cents);

-- No paid order may lack a payment reference.
SELECT count(*)
FROM orders
WHERE status = 'paid' AND payment_reference IS NULL;

Also compare row counts and age distributions, run application smoke tests, verify schema compatibility, and sample recent high-value entities. “Postgres accepts connections” is necessary and radically insufficient.

Measure actual recovery time

An RTO in a document is an aspiration. A timed restore is evidence.

Break the exercise into stages:

detection → decision → environment → data restore
→ validation → reconciliation → traffic → stable operation

Record p50 and worst observed duration for each stage. This exposes the real bottleneck. Teams often buy faster storage while approval, credentials, DNS, or validation consumes most of the outage.

Run at least three kinds of drills:

  1. Routine restore: automated restore into an isolated environment.
  2. Scenario exercise: region loss, credential compromise, or operator deletion.
  3. Unannounced execution: a bounded exercise proving the runbook works without its author.

If the same person writes, operates, and validates the procedure, key-person risk remains untested.

Reconcile the gap

Recovery produces a point in time, not necessarily a final truth. External systems may have accepted effects after the restored point: payments settled, emails sent, devices acted, or partners recorded webhooks.

For every external effect, define:

  • the system of record;
  • a stable idempotency key;
  • how to query the external outcome;
  • who resolves ambiguity;
  • how evidence is retained.

The last mile of recovery is convergence. Without reconciliation, a technically successful restore can create a financially incorrect product.

CTO review

Ask for evidence, not reassurance:

  1. Which business capability does each RPO and RTO protect?
  2. When was the last full restore, and what duration was observed?
  3. Can recovery proceed if the primary cloud account is unavailable?
  4. Which dependencies must be restored in which order?
  5. What prevents restored workers from repeating external effects?
  6. Which invariant queries decide that traffic may return?
  7. Who has successfully executed the runbook besides its author?

Backups are inventory. Recoverability is a repeatedly demonstrated operating capability.

References

>