Serializable Isolation Is a Retry Protocol, Not a Checkbox

Most teams reach for a row lock when concurrent requests break an invariant. That works when the rows that must be locked are already known. It fails for predicates such as “at least one doctor must remain on call” because two transactions can update different rows while jointly violating the rule.

The production invariant is:

Every committed transaction must be explainable as part of one serial execution, and any rejected transaction must be safe to run again from the beginning.

PostgreSQL’s SERIALIZABLE level provides that first property through Serializable Snapshot Isolation. The application must provide the second.

The anomaly is between decisions, not rows #

Assume two doctors are on call. Each transaction checks that another doctor remains and then removes its own doctor:

BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT count(*) FROM doctor_shift WHERE on_call = true;
UPDATE doctor_shift SET on_call = false WHERE doctor_id = $1;
COMMIT;

The transactions update different rows, so ordinary row-level conflict detection does not connect them. Both can observe a count of two and commit, leaving zero doctors.

Under SERIALIZABLE, PostgreSQL tracks read/write dependencies, including predicate reads. If the concurrent history cannot be serialized safely, one transaction aborts with SQLSTATE 40001. That abort is not database instability. It is the database refusing to certify an impossible history.

Retry the decision, not the final statement #

A serialization failure invalidates every decision made from the old snapshot. Retrying only COMMIT or the last UPDATE preserves stale reasoning.

for (let attempt = 1; attempt <= 4; attempt++) {
  const client = await pool.connect()
  try {
    await client.query('BEGIN ISOLATION LEVEL SERIALIZABLE')
    const result = await changeOnCallState(client, command)
    await client.query('COMMIT')
    return result
  }
  catch (error) {
    await client.query('ROLLBACK').catch(() => {})
    if (sqlState(error) !== '40001' || attempt === 4)
      throw error
    await sleep(randomBetween(5, 25) * attempt)
  }
  finally {
    client.release()
  }
}

The callback must contain the complete read-decide-write unit. It should not send email, call a payment provider, or publish a message directly: a retry could repeat those effects. Commit an outbox record in the same transaction and deliver it separately with stable event identity.

Retrying forever hides overload #

Serializable transactions can abort more often when they are long, touch broad predicates, or contend on hot data. A retry loop converts some aborts into latency and additional load. Bound it by both attempts and the request deadline.

Measure at least:

• serialization failure rate by transaction type;
• attempts required before success;
• time spent waiting before retry;
• exhausted retries and caller deadlines;
• the predicates or aggregates responsible for contention.

When failures become routine, the answer is rarely a larger retry limit. Shorten the transaction, reduce the rows it examines, partition the hot invariant, or deliberately serialize access with a lock or queue.

Know the failure classes #

PostgreSQL recommends retrying serialization failures (40001) and, in some applications, deadlocks (40P01). A unique violation is not automatically retryable: it may be the correct business result. A connection failure during commit is ambiguous and requires idempotency or reconciliation, not blind repetition.

The CTO decision #

Use serializable isolation for invariants spanning a predicate or multiple rows when the cost of an invalid state exceeds the cost of occasional aborts. Do not apply it as a global badge of correctness. Establish a transaction boundary, idempotency model, retry budget, and contention SLO together.

The database can reject unsafe histories. Only the application can decide whether repeating the business command is safe.

References #

• PostgreSQL: Transaction Isolation
• PostgreSQL: Serialization Failure Handling
• PostgreSQL: Explicit Locking
• Snippet: Retry a PostgreSQL serializable transaction