A Kafka partition can be processed by only one member of a consumer group at a time. When members join, leave, stall, or when partition counts change, the group must transfer ownership. That transfer is a distributed coordination protocol with correctness boundaries—not a harmless restart detail.
The invariant is:
At any group generation, each partition has at most one valid owner, and a new owner starts only from an offset consistent with the previous owner’s completed effects.
Eager rebalancing revokes everything #
In the classic eager protocol, consumers revoke all partitions, the coordinator calculates a new assignment, and consumers resume. Even partitions whose owners do not change stop processing.
steady state → member change → revoke all → assign all → restore state → resumeThat pause becomes expensive when consumers maintain caches, local state stores, database connections, or large in-flight batches. Frequent deployments, autoscaling, long garbage-collection pauses, and processing that exceeds poll limits can create a rebalance loop where the group spends more time coordinating than consuming.
Cooperative rebalancing narrows movement #
CooperativeStickyAssignor preserves as many assignments as possible and transfers only partitions that must move. Ownership changes happen incrementally: a consumer first revokes a subset, then the next round assigns those partitions elsewhere.
This reduces disruption, but rollout must be compatible across the group. Kafka documentation requires all consumers to support the cooperative assignor. A mixed strategy can fall back to the common protocol selected by the group. Treat changing assignors like a protocol migration, not a one-client configuration tweak.
Kafka 4.x also provides the next-generation consumer rebalance protocol (KIP-848), enabled with group.protocol=consumer. Assignment moves server-side and heartbeat/session settings are coordinated differently. The migration choice must match broker and client versions; copying old timeout tuning into the new protocol without checking ownership is unsafe.
Offset commit is not effect commit #
The dangerous boundary is between processing a record and committing its next offset.
read record 42
write invoice to database
crash before committing offset 43
new owner starts at 42
write invoice againRebalancing makes this ambiguity visible, but does not create it. Automatic offset commits can acknowledge work before external effects finish. Committing after the effect produces at-least-once delivery and therefore requires idempotency. Kafka transactions can atomically combine consumed offsets with records produced to Kafka, but cannot atomically include an arbitrary external database.
Use stable event identity at the destination:
INSERT INTO processed_event(event_id, processed_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING;The business mutation and deduplication record must share one database transaction.
Polling is a lease on ownership #
Consumers must continue polling to prove liveness and receive group events. If processing blocks the poll loop beyond max.poll.interval.ms, the coordinator can remove the member and assign its partitions elsewhere. The old process may still be finishing work even though its ownership is gone.
Separate polling from bounded processing, pause partitions under backpressure, and cap in-flight work. Do not solve slow handlers by setting an enormous poll interval; that also increases the time before genuinely dead processing is recovered.
Static membership with group.instance.id can reduce reassignment after brief restarts by giving an instance a stable identity. It does not make two processes with the same identity safe. Deployment systems must prevent overlapping replicas that claim one member identity.
Revoke callbacks are a deadline #
On revocation:
- stop admitting records from the revoked partitions;
- finish or cancel bounded in-flight work;
- persist required state and offsets;
- release partition-scoped resources;
- return before the rebalance timeout.
If cleanup depends on a slow external system, the group can stall. Prefer restartable state and idempotent effects over heroic shutdown hooks.
Operational evidence #
Track rebalance rate and duration, assigned partitions per member, time since last poll, records in flight, revoke-handler duration, consumer lag, duplicate suppression, offset-commit failures, and generation-related errors. Correlate lag spikes with deployments and autoscaling events.
The CTO decision #
Select the rebalance protocol from the cost of moving ownership. Use cooperative assignment and static identities to reduce unnecessary movement, but preserve correctness with bounded processing and idempotent effects. Scale on sustained lag and processing capacity, not on momentary queue depth that causes members to churn.
Kafka gives the group one owner at a time. Your application still has to ensure that the old owner’s ambiguous effects and the new owner’s retries converge.