kafka-flow master at 1a062dc, kafka-clients 4.3.1.
Summary
This affects CooperativeStickyAssignor on the classic protocol, where retained partitions keep writing through a rebalance. Eager assignors (StickyAssignor, RangeAssignor) are not affected: they tear every flow down before the generation bumps.
In cachingTransactional mode every snapshot flush and periodic offset commit is a Kafka transaction bound to the consumer's group generation (KIP-447). When the broker rejects one for a stale generation, kafka-flow fails the flow. The rejection already is the fence: the transaction aborted, nothing landed, the producer stays usable, and the consumer refreshes its generation on the next completed rebalance. Failing the flow adds a leave and a rejoin that bump the generation again and fence the peers' in-flight transactions. Under a rolling deploy this becomes a self-sustaining restart cascade.
Why a still-valid owner gets fenced
The coordinator checks member and generation, not partitions (ClassicGroup.validateOffsetCommit). The generation advances on every completed rebalance, even one that changes nothing for this member. kafka-flow reads it after each poll, so a transaction started between the new generation forming and the next post-poll refresh carries the old one and is rejected. Under the cooperative assignor retained partitions keep committing during a rebalance, so this window is hit routinely. The design doc names it: "the retained partition's next transactional commit would be spuriously fenced though the member still owns it".
What kafka-flow does with the rejection
GroupCommit.commitBatch aborts and fails every pending item (KafkaSnapshotWriteDatabase.scala#L135-L158). Both callers fail the flow:
Why failing is redundant
The producer survives (commitBatch already aborts). The consumer heals on its own: a fenced member completes the rebalance round on its next poll() and keeps its retained partitions. Nothing was written: the key's state is still in memory and the next tick flushes it under the refreshed generation.
How the failure amplifies
Closing the consumer sends LeaveGroup, which opens a rebalance and bumps the generation, fencing every peer with a transaction in flight. The retry joins, usually a second rebalance and a second bump. Each fenced peer repeats the cycle. Every restart also re-runs partition recovery, so the cost grows with the snapshot topic.
The existing escape and its defect
ignorePersistErrors keeps the flow alive through a fenced flush but tolerates every persist error, so a genuine fault stalls the committed offset silently. It covers neither unloadOrphaned nor the periodic commit. In persistPeriodicallyAndUnloadOrphaned (TimerFlowOf.scala#L189-L198) it also removes the key whether or not its persist succeeded, so the committed offset can advance past state that was never persisted.
Related
kafka-flow
masterat 1a062dc, kafka-clients 4.3.1.Summary
This affects
CooperativeStickyAssignoron the classic protocol, where retained partitions keep writing through a rebalance. Eager assignors (StickyAssignor,RangeAssignor) are not affected: they tear every flow down before the generation bumps.In
cachingTransactionalmode every snapshot flush and periodic offset commit is a Kafka transaction bound to the consumer's group generation (KIP-447). When the broker rejects one for a stale generation, kafka-flow fails the flow. The rejection already is the fence: the transaction aborted, nothing landed, the producer stays usable, and the consumer refreshes its generation on the next completed rebalance. Failing the flow adds a leave and a rejoin that bump the generation again and fence the peers' in-flight transactions. Under a rolling deploy this becomes a self-sustaining restart cascade.Why a still-valid owner gets fenced
The coordinator checks member and generation, not partitions (
ClassicGroup.validateOffsetCommit). The generation advances on every completed rebalance, even one that changes nothing for this member. kafka-flow reads it after each poll, so a transaction started between the new generation forming and the next post-poll refresh carries the old one and is rejected. Under the cooperative assignor retained partitions keep committing during a rebalance, so this window is hit routinely. The design doc names it: "the retained partition's next transactional commit would be spuriously fenced though the member still owns it".What kafka-flow does with the rejection
GroupCommit.commitBatchaborts and fails every pending item (KafkaSnapshotWriteDatabase.scala#L135-L158). Both callers fail the flow:attemptToPersistre-raises unlessignorePersistErrorsis set (TimerFlowOf.scala#L233-L246);unloadOrphanedfails regardless of the flag (TimerFlowOf.scala#L60-L66).PartitionFlowschedules the commit without error handling (PartitionFlow.scala#L283-L285).Why failing is redundant
The producer survives (
commitBatchalready aborts). The consumer heals on its own: a fenced member completes the rebalance round on its nextpoll()and keeps its retained partitions. Nothing was written: the key's state is still in memory and the next tick flushes it under the refreshed generation.How the failure amplifies
Closing the consumer sends
LeaveGroup, which opens a rebalance and bumps the generation, fencing every peer with a transaction in flight. The retry joins, usually a second rebalance and a second bump. Each fenced peer repeats the cycle. Every restart also re-runs partition recovery, so the cost grows with the snapshot topic.The existing escape and its defect
ignorePersistErrorskeeps the flow alive through a fenced flush but tolerates every persist error, so a genuine fault stalls the committed offset silently. It covers neitherunloadOrphanednor the periodic commit. InpersistPeriodicallyAndUnloadOrphaned(TimerFlowOf.scala#L189-L198) it also removes the key whether or not its persist succeeded, so the committed offset can advance past state that was never persisted.Related