A poison message is a record that repeatedly fails under the current consumer implementation. It may contain malformed bytes, an unsupported schema, an impossible domain transition, or a value that exposes a code defect. Calling every failure “transient” lets one record become an indefinite operating condition.
The difficult decision is what happens to records behind it. A dead-letter queue preserves a failed record somewhere else. It does not establish whether later records are safe to process, whether the consumer may advance its offset, or whether anyone will repair the abandoned business operation.
This article proposes a recovery policy for a conventional Kafka consumer group. The API references are for Kafka 4.1 Java clients; confirm configuration behavior against the version and group protocol you deploy.
Begin with the business sequence #
Suppose one partition contains:
offset 120: OrderCreated(order-7)
offset 121: OrderAddressChanged(order-7) <- consumer fails
offset 122: OrderDispatched(order-7)
offset 123: OrderCreated(order-9)Skipping 121 may let a parcel leave for the wrong address. Blocking the partition preserves that sequence but also delays order-9, which may be unrelated.
Neither behavior is universally correct. Identify whether ordering is required across the whole partition, within one business key, or only between specific event types. Kafka’s partition organization is a transport choice; the domain determines which reorderings are acceptable.
Write the invariant before the retry configuration: a dispatch must not execute while an earlier required address change for that order is unresolved.
Separate four failure classes #
A temporary dependency outage should trigger controlled backoff and reduced admission. Retrying malformed bytes against the same parser will not repair them. An unknown schema may require a compatible deployment. A business rejection may be an expected terminal result.
Keep these classifications visible:
| Failure | Candidate policy | Evidence required to resume |
|---|---|---|
| Dependency timeout | Pause affected work; retry within a budget | Dependency can complete useful work |
| Unsupported schema | Quarantine or stop, according to ordering needs | Compatible consumer and a replay test |
| Invalid business transition | Record rejection or request repair | Domain owner defines the correct outcome |
| Unknown consumer defect | Contain and investigate | Reproducer, fix, and regression test |
Do not send every timeout to a dead-letter topic. During an outage that converts infrastructure failure into a large backlog of manual repair.
The offset is an acknowledgment boundary #
The Kafka consumer API distinguishes the current position from the committed position used for recovery. Committed offsets identify where consumption should resume, conventionally the next record to process.
If records are dispatched concurrently, completion order may differ from partition order. Suppose 120 finishes, 121 is still running, and 122 finishes. Committing 123 would skip 121 on restart.
Maintain a contiguous completion frontier per partition. Advance it only when every earlier record has reached an accepted durable outcome. If quarantine counts as completion, make that a deliberate policy with a durable acknowledgment, not an exception handler that logs and continues.
The recovery design must also respect partition ownership during rebalances. An old worker must not independently acknowledge work after ownership has moved.
Quarantine is a write path #
A useful quarantine envelope records original identity and enough evidence to replay safely:
{
"source": {
"topic": "orders",
"partition": 2,
"offset": 121
},
"eventId": "evt-address-7",
"consumerVersion": "orders-handler-v3",
"failureClass": "unsupported_schema",
"schemaVersion": 8,
"payloadReference": "restricted-quarantine/evt-address-7",
"repairStatus": "pending"
}This is a proposed application envelope, not Kafka configuration. Restrict payload access and retention independently from operational metadata. A dead-letter topic can otherwise become a second, less governed customer database.
There is a crash window between acknowledging quarantine and committing the source offset. With ordinary separate writes, a restart can produce another quarantine entry. Make quarantine identity deterministic, such as source topic, partition, and offset, and tolerate duplicate delivery.
For Kafka-to-Kafka processing, transactions can couple output publication and consumed offsets when configured correctly. They do not make an external database mutation or email part of the same atomic outcome. Review the side-effect boundary before claiming exactly-once recovery.
Choose the price of preserving order #
Three designs are worth comparing.
A partition stop is easy to reason about but has a large delay radius when unrelated keys share the partition. A per-key holding area can release unrelated work, but adds durable sequencing state, storage, recovery logic, and memory limits. Processing everything after quarantine is operationally simple only when the domain explicitly tolerates missing predecessors.
There is no free bypass. If the system needs per-key holds, decide who owns the blocked-key registry and how a new consumer reconstructs it. Never rely only on an in-memory set that disappears during a rebalance.
This is also why adding partitions is not a neutral throughput change. See partition changes and ordering contracts.
Keep the consumer alive without pretending progress #
Kafka exposes controls for polling and automatic commits in its consumer configuration. Long processing can interact with the poll interval and group ownership. Pausing a partition does not remove the need to service the consumer appropriately.
Design retries so the poll loop and worker ownership remain understandable. Do not block one consumer thread indefinitely inside a failing handler. A green process-health check is weak evidence when one business key has been blocked for hours.
Measure the age of the oldest unresolved operation, quarantine arrival rate, repair completion rate, duplicate replay rate, and the count of blocked keys. Consumer lag alone cannot reveal whether the business has recovered.
Replay is a release #
Replaying a repaired record is another production change. Preserve its original event identity, record the repair version, and prevent an operator from accidentally replaying the same effect repeatedly.
Test these crash points: before quarantine durability, after quarantine but before source acknowledgment, during a rebalance, and after the side effect but before completion is recorded. Add the original failing payload as a restricted regression fixture when policy permits.
The final approval question is concrete: if this record is isolated, which later actions remain valid? A dead-letter queue is useful when the team can answer that question and can bring the operation back through a controlled repair path.