Kafka Exactly-Once Stops at the Side-Effect Boundary

“Exactly once” is useful only after naming exactly which state changes share a transaction.

Kafka can atomically commit consumed offsets, state-store changes, and records produced to Kafka. That is a powerful boundary. It does not automatically include a payment gateway, email provider, warehouse, or arbitrary database.

The invariant is narrower than the slogan:

one committed Kafka input
→ one visible Kafka/state-store result

The moment a handler crosses into a system that does not participate in the Kafka transaction, the outcome can become ambiguous.

The crash window still exists #

Consider a consumer that charges a card and then commits its offset:

1. consume order
2. payment provider accepts charge
3. process crashes before offset commit
4. order is consumed again

Kafka correctly redelivers because it cannot know whether the external call succeeded. Retrying without a stable idempotency key can charge twice. Committing before the call merely reverses the failure: a crash can lose the charge.

No ordering removes the uncertainty. The boundary needs a protocol.

Choose the protocol by ownership #

If the durable state is in your relational database, use a local transaction to write both domain state and an outbox record. A relay publishes the outbox record later. Consumers remain idempotent because delivery can repeat.

If the result remains inside Kafka, transactions and read_committed consumers can keep offsets and output records atomic.

If the effect belongs to an external provider, send a stable operation identity and reconcile:

await payments.charge({
  idempotencyKey: `order:${order.id}:capture:v1`,
  amount: order.total,
})

The provider must actually persist and enforce that key. A UUID generated on every retry is not idempotency.

Consumer policy #

Classify failures rather than retrying everything. Validation failures belong in a terminal path. Transient infrastructure failures may retry with a deadline and jitter. Ambiguous writes require a status lookup or reconciliation job—not blind repetition.

Record four identities:

• business operation ID;
• source topic, partition, and offset;
• external provider request ID;
• resulting domain-state version.

Those identifiers make an incident reconstructable. Without them, “exactly once” becomes a dashboard label that cannot answer whether value moved.

Production review #

Ask these before enabling exactly-once processing:

1. Which writes are inside the transaction coordinator?
2. Which side effects remain outside?
3. What happens after success but before acknowledgement?
4. Can the same business operation arrive through another channel?
5. How is a stuck or ambiguous operation reconciled?
6. Are transaction timeouts and consumer rebalances visible?

Kafka transactions have operational costs and configuration requirements. Use them when atomic Kafka state is the requirement, not as a substitute for defining business idempotency.

Test the acknowledgement gaps #

Happy-path integration tests do not exercise the guarantee. Inject a crash after the external provider accepts a request but before the consumer commits. Repeat after the database commits but before an outbox relay publishes. Restart a producer after its transaction times out. In each case, assert the final business state and the evidence available to an operator.

Also test poison records. Retrying a deterministic validation failure can block a partition and hide useful work behind it. A terminal path must preserve the original record, failure reason, code version, and replay authorization without creating an ungoverned “dead-letter queue forever.”

Conclusion #

Exactly-once semantics are not false; they are scoped. Good architecture writes that scope beside the promise.

Keep atomic work inside one coordinator. Where that is impossible, make repetition safe, preserve operation identity, and build reconciliation as a first-class path. The honest guarantee is often not “the message runs once,” but “the business effect converges to one defensible outcome.”

Further reading: Apache Kafka documentation, Kafka Streams core concepts, transactional outbox guarantees, and Kafka rebalance coordination.