Kafka Tombstones: Why a Compacted Topic Can Leave Stale Consumer State

A service resumes consuming a compacted Kafka topic after a long outage. Its offset advances, its lag eventually reaches zero, and its local database still contains a customer that was deleted while it was offline. The consumer is caught up in position, but not correct in state.

A tombstone is a deletion signal with a retention policy, not permanent evidence that every downstream copy has been erased. If your consumer can outlive that signal, you need an explicit rebuild or reconciliation procedure.

The failure matters anywhere a log feeds a serving copy: account directories, entitlement caches, routing tables, and search indexes. Position, freshness, and correctness are different properties. This is the operational edge of a CDC recovery contract, not merely a storage optimization.

What compaction preserves #

For a keyed compacted log, Kafka can remove superseded values while retaining the latest state for a key. A record with a non-null key and a null value is a tombstone. Older values and, eventually, the tombstone itself can be removed. Compaction preserves surviving records’ offsets and order; it does not promise one physical record per key at every instant. Kafka log-compaction design.

Neither the JSON string "null" nor an object such as { "deleted": true } is automatically a Kafka tombstone. Those are ordinary non-null payloads unless your application assigns them special meaning. Validate the serializer and consumer contract rather than relying on what a console viewer displays.

Follow one key through the outage #

Here is an illustrative history, not an observed production incident:

offset 100: account-7 -> {enabled: true}
             consumer stores account-7, then goes offline
offset 140: account-7 -> null
             deletion occurs; consumer remains offline
later:       cleaner removes obsolete value and tombstone
             consumer resumes, but its old local row remains

The resumed consumer cannot apply a deletion record that is no longer available. An offset gap is not a list of deleted keys; compaction can create gaps for other reasons. Inferring deletions from missing offsets would corrupt unrelated state.

A fresh projection starts without the obsolete row. A stale projection already contains it. That distinction explains why “we replayed everything still available” is not a sufficient repair statement when replay means merging into an old database.

Define a recovery window, not a lucky default #

Kafka’s delete.retention.ms controls tombstone retention. The documentation also describes a bound for consumers scanning from the beginning to obtain a valid final-state snapshot. Do not interpret it as an exact wall-clock deletion appointment or an unlimited recovery guarantee. Combining compact with delete also allows time/size-based removal of old segments. Topic configuration.

My operational policy would compare four durations:

outage detection
+ incident response
+ consumer catch-up or full rebuild
+ safety margin

This is a risk worksheet, not a formula that overrides Kafka’s cleaner semantics. A consumer with unbounded downtime cannot be protected by any finite retention choice. Increasing retention buys recovery time and spends storage; it does not eliminate the need for a recovery boundary.

Record the maximum supported interruption in the service contract. When you can no longer establish that missed deletions remain recoverable, stop treating the existing projection as authoritative. This may mean removing it from service rather than advertising a green lag metric.

Apply events with an unambiguous reducer #

The smallest useful example distinguishes empty strings from actual null values:

function applyRecord(state, key, value) {
  if (typeof key !== 'string' || key.length === 0)
    throw new Error('Expected a non-empty application key')
  if (value === null) {
    state.delete(key)
    return
  }
  if (typeof value !== 'string')
    throw new Error('Expected a decoded string or null tombstone')
  state.set(key, value)
}

This in-memory reducer is intentionally not a consumer implementation. A real sink must coordinate its state changes and offset bookkeeping so a crash does not acknowledge unapplied work. Preserve per-key ordering, and define what happens when decoding fails. See Kafka poison-message policy before skipping a problematic record.

The reducer makes another limit visible: applying no records cannot remove an existing key. No amount of retry logic inside this function repairs an already-missed tombstone.

Rebuild into a new generation #

A safer rebuild design creates a new, empty projection rather than merging into the old one. Capture per-partition target offsets, replay within the supported retention window, and keep consuming new events. Validate key counts and representative invariants before moving readers to the new generation.

Those offsets are progress targets, not automatically a globally atomic business snapshot. If your product requires cross-partition consistency, define a stronger snapshot-and-log protocol or coordinate with the authoritative source.

Keep the old projection available for investigation, but do not merge its unexplained rows back into the repaired one. Where a full rebuild is too expensive, maintain an authoritative deletion ledger or run source-to-sink reconciliation. Those are additional systems with their own retention, access, and failure policies.

The runbook should distinguish these failures #

Monitor the age of unapplied work, rebuild duration, and sink correctness checks—not only offset lag. A low event rate can make a small lag numerically reassuring while an important deletion remains unhandled.

The architecture decision is whether this topic is a reconstructable current-state feed, a durable historical record, or both through separate storage. Treating it as all three without a retention contract is how a small optimization becomes a recovery surprise. Continue with data-retention architecture and systems architecture.

References #

• Kafka design: log compaction and its guarantees
• Kafka topic configuration: cleanup and tombstone retention