PostgreSQL Logical Replication Failover: The Slot Is Part of Your Recovery Plan

Aug 20

By

postgresqldatabase engineeringdistributed systemsdisaster recovery

Logical replication is often added for a migration, an analytics store, or a regional read model. The happy path looks deceptively clean: publish changes, apply them elsewhere, watch lag approach zero. The recovery path is harder.

If the publisher fails over but its logical replication slot does not, the subscriber’s continuity contract disappears with the old primary. A healthy standby is therefore not sufficient. The replication state that tells the publisher what the subscriber still needs must also be recoverable.

Separate the three state machines

Treat logical replication as three related but independent state machines:

publisher database state
        |
        +--> logical slot position and retained WAL
        |
        +--> subscriber apply position and local state

Physical replication protects database pages and WAL. A logical slot protects a consumer position. The subscriber maintains its own apply progress. A failover is safe only when the promoted standby has a usable copy of the slot and is far enough ahead for the subscriber to continue.

PostgreSQL’s current documentation supports synchronizing failover-enabled logical slots to a physical standby. A subscription created or altered with failover = true identifies the logical slot as one that must survive publisher failover. But synchronization is asynchronous: configuration is not proof of readiness.

The readiness invariant

Before a planned promotion, prove all of the following:

  1. the standby has received and replayed the WAL needed for the logical slot;
  2. the corresponding slot exists on the standby and reports synchronized readiness;
  3. the subscriber is not ahead of the slot state available on the standby;
  4. the connection endpoint will resolve to the promoted publisher;
  5. the subscriber can resume without recreating the subscription or losing its origin state.

PostgreSQL explicitly warns that the standby must be ahead of the subscriber because slot synchronization happens asynchronously. This converts “HA is configured” into a measurable gate.

An operating sequence

For a planned failover:

freeze topology changes
  -> inspect publisher/subscriber lag
  -> verify failover slots synchronized on target standby
  -> stop or fence the old primary
  -> promote standby
  -> move publisher endpoint
  -> observe subscriber resume and converge

Do not begin with the promotion command. Begin with evidence and fencing. A network partition can leave the old primary writable; logical changes from two primaries cannot be reconciled by wishful thinking.

The exact catalog fields vary by PostgreSQL version, so build checks from that version’s documentation rather than pasting a query from an old runbook. At minimum, alert on slot activity, retained WAL, subscriber lag, apply errors, and synchronization state on every eligible failover target.

WAL retention is a capacity risk

A logical slot prevents removal of WAL still required by its consumer. When a subscriber stalls, storage consumption can grow until it becomes the incident. High availability therefore couples two budgets:

  • continuity budget: retain enough WAL for subscribers to recover;
  • storage budget: prevent an abandoned consumer from retaining WAL indefinitely.

Set an explicit maximum acceptable lag in bytes and time. Alert before storage headroom is exhausted. Decide whether an over-budget subscriber is paused and rebuilt, or whether the publisher is allowed to keep retaining WAL. There is no universally correct answer; there must be an owner and a threshold.

Replication does not copy every database behavior

Logical replication transfers table data changes. It is not a full database clone. PostgreSQL documents important restrictions: schema definitions are not replicated, sequence state is not replicated, and large objects are not replicated. DDL must be coordinated separately, and the subscriber schema must remain compatible with incoming rows.

That creates a deployment order:

expand subscriber schema
 -> expand publisher schema
 -> deploy compatible writers
 -> allow new data shape
 -> validate replication
 -> contract later

If the publisher begins emitting a column the subscriber does not understand, availability at the database layer cannot rescue the apply pipeline.

Sequence divergence matters during cutover. If a subscriber becomes the new write primary, initialize its sequences to values that cannot collide with replicated keys. For globally distributed writers, prefer an ID scheme whose uniqueness does not depend on a single local sequence.

Recovery objectives must name the consumer

“The database has a 30-second RPO” is incomplete. Which state?

  • Physical RPO: how much primary database state can be lost?
  • Logical-consumer RPO: how far behind can the subscriber be?
  • Apply RTO: how long until it resumes after publisher promotion?
  • Rebuild RTO: how long to recreate the subscriber if continuity fails?

Analytics may tolerate a rebuild. A payment ledger read model may not. Apply one recovery objective to every logical consumer and the architecture will either waste money or hide risk.

Failure drills that reveal the truth

Run at least these tests away from customer traffic before trusting the topology:

  1. planned promotion while the subscriber is caught up;
  2. promotion while the subscriber has controlled lag;
  3. subscriber outage long enough to exercise WAL-retention alarms;
  4. schema mismatch that stops apply, followed by repair and resume;
  5. endpoint failure that proves retry and DNS behavior;
  6. old-primary fencing failure to validate split-brain controls.

Record time to detection, time to resume, duplicate or missing rows, retained WAL, and manual steps. A runbook that has never consumed its own telemetry is documentation, not recovery capability.

CTO review checklist

Ask the team:

  • Are logical slots synchronized to every eligible promotion target?
  • What automated condition prevents promotion when the slot is not ready?
  • How much WAL can a stalled subscriber retain before we intervene?
  • Who coordinates subscriber-compatible DDL?
  • How are sequences and non-replicated objects handled at cutover?
  • Have we measured resume time under real lag?

Logical replication failover is not a checkbox beside physical HA. The slot, subscriber position, endpoint, schema, and fencing mechanism form one recovery system. Operate them as one.

References

>