Durable execution can recover orchestration after a worker crash by replaying an event history. It cannot reach into a payment provider, email server, or model API and make an ambiguous external effect exactly once.
The invariant is:
Workflow decisions replay deterministically; every external effect is independently safe to repeat or reconcile.
Confusing those two guarantees creates workflows that resume reliably and duplicate money just as reliably.
Replay restores decisions #
In a replay-based engine such as Temporal, the service persists workflow events. A worker re-executes workflow code and matches generated commands against recorded history. Completed activity results are supplied from history rather than invoking the activity again.
This requires workflow code to be deterministic. Wall-clock reads, randomness, network calls, and mutable process state can produce a different command sequence during replay. Use workflow-provided clocks and randomness, and move external I/O into activities.
workflow code: deterministic orchestration and policy
event history: durable decisions and activity outcomes
activities: non-deterministic external effectsThe activity boundary is still ambiguous #
Consider an activity that charges a card:
- the provider commits the charge;
- the worker loses its connection before reporting completion;
- the engine sees no completion event;
- the activity is retried.
No orchestration engine can infer the provider’s state from a missing acknowledgement. The activity needs a stable idempotency key derived from workflow and business command identity.
await payments.charge({
orderId,
amount,
idempotencyKey: `charge:${orderId}`,
})If the provider lacks idempotency, record an intent locally, execute with a durable operation ID where possible, and reconcile ambiguous results before another attempt.
Timeouts describe different failures #
One timeout cannot express queue delay, execution time, and the total retry horizon. A production activity policy separates:
| Boundary | Meaning |
|---|---|
| schedule-to-start | worker capacity or task-queue delay |
| start-to-close | one attempt exceeded its execution budget |
| schedule-to-close | the complete activity, including retries, is no longer useful |
| heartbeat | a long activity stopped proving progress |
Retry only failures classified as transient. Authorization, validation, and policy denial should be terminal. Backoff and jitter protect the unhealthy dependency; maximum attempts or total timeout protect the business deadline.
History is not a data lake #
The event history is operational state. Large payloads, token streams, and unbounded loops inflate replay cost and can reach platform history limits. Temporal documents event-count and signal/update limits and provides Continue-As-New to start a fresh history while carrying forward compact state.
For an AI agent, store large transcripts and artifacts in an external durable store. Put references, tool decisions, approvals, and compact outcomes in workflow state. Model calls and tool calls belong in activities because they are non-deterministic and may have external cost or side effects.
Deployment requires replay compatibility #
A workflow can live longer than one application release. Changing its code may cause old histories to produce new commands during replay. Use the platform’s versioning mechanisms or pin compatible workers. Test replay against sampled production histories before rollout.
The deployment question is not only “does the new code pass unit tests?” It is “does the new code still explain every active execution’s past?”
Production review #
- Is workflow code deterministic?
- Does every activity have a stable business identity?
- Are retryable failures explicitly classified?
- Do timeouts reflect queue, attempt, and total lifetime separately?
- Can long activities heartbeat and resume from checkpoints?
- Is history growth bounded?
- Can the new worker replay old histories?
- Can operators pause, retry, terminate, and audit an execution safely?
The CTO decision #
Use durable workflows when a business process spans failures, time, human input, or several services and deserves a first-class execution record. Do not use them to disguise poorly defined side effects. Durable orchestration reduces state-machine plumbing; idempotency and reconciliation still protect the world outside the engine.