The Transactional Outbox Is Not the Delivery Guarantee

Aug 20

By

distributed systemsbackend engineeringevent drivenreliability

The transactional outbox solves one precise problem: it makes a business-state change and the creation of its integration event part of the same database transaction. That is important. It is not the same as guaranteeing that every downstream effect happens exactly once.

The production question is not “do we have an outbox?” It is: what invariant survives every crash boundary from command to consumer side effect?

Name the dual-write failure

Suppose checkout must update an order and publish OrderConfirmed.

write order -> crash -> publish event       (missing event)
publish event -> crash -> write order       (phantom event)

No ordering of two independent writes removes both windows. A distributed transaction can coordinate some systems, but it adds support and availability constraints most service architectures do not want.

The outbox places both records in one local transaction:

BEGIN;

UPDATE orders
SET status = 'confirmed', version = version + 1
WHERE id = $1 AND status = 'pending';

INSERT INTO outbox (
  event_id, aggregate_type, aggregate_id,
  aggregate_version, event_type, payload, created_at
) VALUES (
  $2, 'order', $1, $3,
  'OrderConfirmed', $4::jsonb, now()
);

COMMIT;

Now the order and event exist together or not at all. AWS Prescriptive Guidance describes this as the core purpose of the pattern and explicitly warns that duplicate delivery still requires idempotent consumers.

Follow every crash boundary

The relay reads committed outbox rows and publishes them to a broker:

database transaction
      -> outbox row
      -> relay publish
      -> broker acknowledge
      -> mark row delivered

If the relay crashes after the broker accepts the event but before delivered_at is stored, it publishes again. If it marks delivered before the broker accepts, the event can be lost. Therefore a conventional relay is at-least-once, and duplicate publication is the correct failure behavior.

Process rows concurrently without letting workers claim the same work:

SELECT event_id, payload
FROM outbox
WHERE delivered_at IS NULL
ORDER BY created_at, event_id
FOR UPDATE SKIP LOCKED
LIMIT 100;

Keep claims short-lived. Do not hold a database transaction open during an unbounded network call. A practical design records a lease, commits, publishes, then marks success; an expired lease makes work recoverable after a worker dies.

Idempotency belongs at the effect

A consumer that “checks then acts” can still race:

check event unseen -> charge card -> crash -> record event seen

The retry charges again. The deduplication record must be atomic with the side effect where possible. For a database-owned effect:

BEGIN;

INSERT INTO consumed_events (consumer, event_id)
VALUES ('invoice-projector', $1)
ON CONFLICT DO NOTHING;

-- Continue only when one row was inserted.
UPDATE invoice_summary
SET paid_cents = paid_cents + $2
WHERE customer_id = $3;

COMMIT;

For an external API, pass a stable idempotency key if the provider supports one. If it does not, “exactly once” is not an honest promise. Design a reconciliation process that detects ambiguous outcomes.

Broker-level transactions can provide strong guarantees inside the broker’s boundary. Kafka’s idempotent producer and transactions, for example, can atomically write Kafka records and offsets in supported workflows. They do not make an unrelated payment processor or email provider transactional with Kafka. State the boundary whenever using the phrase exactly once.

Ordering needs a domain, not a global queue

Most systems do not need total order across every event. They need order per aggregate—events for one order, account, or device.

Include aggregate_id and a monotonically increasing aggregate_version. Partition the broker by aggregate ID. Consumers then reject stale versions and detect gaps:

received version 41, stored version 40 -> apply
received version 40, stored version 41 -> duplicate/stale
received version 43, stored version 40 -> gap; defer and investigate

Global ordering reduces parallelism and still does not define business semantics across unrelated entities. Buy only the ordering the invariant requires.

Event payloads are long-lived APIs

An outbox couples the business transaction to an immutable historical record. Treat payload evolution as API evolution:

  • include an event type and schema version;
  • prefer facts (OrderConfirmed) over imperative commands (SendConfirmationEmail);
  • include the minimum context needed for a stable contract;
  • avoid leaking a full internal row that changes whenever the database changes;
  • make consumers tolerate additive fields;
  • retain fixtures for older payload versions.

Replays expose every lazy schema decision. If rebuilding a read model requires bespoke transformations for undocumented events, the event log is not an operational asset.

Operate lag, not just queue depth

Measure the system at each boundary:

  • age of the oldest unpublished outbox row;
  • rows created versus successfully published;
  • relay lease expirations and retry counts;
  • broker consumer lag by partition;
  • duplicate rate at each consumer;
  • dead-letter age and owner;
  • end-to-end time from business commit to visible side effect.

Queue depth alone cannot distinguish a healthy burst from a poison event blocking one aggregate.

Keep outbox cleanup separate from delivery. Delete or archive only rows whose retention window has elapsed and whose delivery evidence is durable. A sudden cleanup job should not compete with the relay on the same hot index.

Common mistakes

Publishing from an ORM hook after commit. It recreates the dual-write window.

Using a random event ID on every retry. The retry becomes a new event and defeats deduplication. Derive or persist the ID with the originating command.

Marking published before acknowledgement. This optimizes for silent loss.

Assuming consumers are idempotent because handlers are short. Test the crash after every external effect.

No replay or poison-event policy. Eventually one malformed payload will stop progress or loop forever.

CTO review checklist

  1. Where is event creation atomic with domain state?
  2. Which crash boundary produces duplicates, and how is each effect deduplicated?
  3. What is the required ordering scope?
  4. How are schema versions evolved and replayed?
  5. What end-to-end metric proves a committed fact became a downstream effect?
  6. Who owns reconciliation when the external outcome is ambiguous?

The outbox is a valuable first link. Reliability comes from connecting the rest: stable identity, recoverable relay state, bounded ordering, idempotent effects, replayable schemas, and observable convergence.

References

>