A database rollback erases uncommitted local changes. A saga compensation creates new business facts after earlier facts have already become visible. Confusing the two produces workflows that look atomic in diagrams and fail messily in production.
The invariant is:
Every committed saga step must have a defined forward recovery, semantic compensation, or explicit manual-resolution state.
Consider order creation, inventory reservation, payment authorization, and shipment. If shipment fails, “undo payment” might mean voiding an authorization, issuing a refund, or creating a receivable. Those outcomes are financially and temporally different.
Model facts, not inverse API calls #
reserve inventory -> authorize payment -> create shipment
X failure
release inventory <- void/refund payment <- mark order resolution_pendingCompensation can fail too. Inventory may already be sold, a refund provider may be unavailable, or a parcel may have left the warehouse. The workflow therefore needs durable state, retry policy, deadlines, and an owner for terminal exceptions.
A useful step record includes:
CREATE TABLE workflow_step (
workflow_id uuid NOT NULL,
step_name text NOT NULL,
operation_id uuid NOT NULL,
state text NOT NULL,
attempt int NOT NULL,
result jsonb,
updated_at timestamptz NOT NULL,
PRIMARY KEY (workflow_id, step_name)
);Each participant must accept a stable operation ID. A timeout creates an ambiguous outcome; the orchestrator should query status or retry idempotently, not assume failure.
Separate technical from business failure #
A transient network error usually calls for bounded forward retry. “Card declined” is a business decision and should not be retried as infrastructure noise. An unknown payment outcome needs reconciliation. Treating all three as exceptions guarantees duplicate effects or stuck workflows.
| Outcome | Action |
|---|---|
| transient dependency failure | bounded retry with deadline |
| deterministic rejection | compensate completed steps |
| ambiguous write | look up by operation ID |
| compensation exhausted | manual-resolution queue |
| deadline exceeded | move to explicit expired state |
Orchestration buys visibility #
Choreography works for a small number of participants, but the workflow becomes implicit across event handlers as branches grow. An orchestrator centralizes state transitions, deadlines, and recovery evidence. It must itself be durable and horizontally safe; “central” should not mean one process with in-memory progress.
Store state transitions before dispatching the next effect, use an outbox to couple state and message creation, and make consumers idempotent. Exactly-once business execution is not supplied by the broker.
Design the human path #
Some effects cannot be compensated automatically. Build an operator view containing the workflow timeline, operation IDs, provider references, current invariant violation, safe next actions, and audit trail. Manual intervention should issue the same idempotent commands as automation rather than editing databases.
Measure age by state, retry and compensation rates, ambiguous outcomes, manual queue size, time to resolution, and money or inventory trapped in intermediate states.
The CTO decision #
Choose sagas only when the business accepts temporary inconsistency and can define recovery for every committed step. Name irreversible boundaries, cap automation, and fund the operator path.
Compensation does not restore history. It moves the business into a new, acceptable state with evidence of how it got there.