Conversation
tobiajo
commented
Jun 27, 2026
tobiajo
force-pushed
the
tj/address-partition-ownership-overlap-possiblity-cassandra-persist-only
branch
15 times, most recently
from
July 1, 2026 20:35
c377f7e to
dfb7869
Compare
tobiajo
added a commit
to tobiajo/kafka-flow
that referenced
this pull request
Jul 3, 2026
Restructure docs/cassandra-single-writer-design.md to focus on why over what: brief mechanics, explicit design choices, and the learnings from the deferred full solution, mirroring the Kafka design doc's shape (Problem, mechanism, Testing, Rejected alternatives, Forward-looking): - Problem now explains why the Kafka fix does not transfer to Cassandra (no transaction to bind the offset commit into, no ownership authority), motivating a per-write, per-key fence. - The mechanism is kept brief and its three load-bearing choices are called out explicitly: per-key granularity, `<=` over `<`, and ordering by data rather than identity. - "Why the store is the whole change" makes the SnapshotFold replay dedup explicit as the property that lets persist-only ship without core changes. - A Compatibility section collects the no-API-change / no-migration / write-side-only-fence consequences. - The deferred full solution (offset-gated deletes, PR evolution-gaming#834) is retold as a forced chain - tombstone, breaking delete(key, offset) API, monotonic buffer, tombstone-floor recovery read, independent events-recovery seeding - with a livelock diagram, the TLA+ results (all defects were liveness, never safety), and the explicit deferral rationale (cost/benefit and KIP-939 timing). - A process note links the stacked PRs (evolution-gaming#828/evolution-gaming#833/evolution-gaming#834/evolution-gaming#835/evolution-gaming#838) so the git history, review conversations and models stay discoverable. Also make the rolling-deploy clock-skew caveat in persistence.md concrete (coordinator vs client write timestamps), and update a test comment referencing a renamed doc section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018jfVxUNrjxpcSFF2hgiar5
tobiajo
force-pushed
the
tj/address-partition-ownership-overlap-possiblity-cassandra-persist-only
branch
12 times, most recently
from
July 5, 2026 09:12
9d443af to
472228d
Compare
tobiajo
force-pushed
the
tj/address-partition-ownership-overlap-possiblity-cassandra-persist-only
branch
21 times, most recently
from
July 8, 2026 13:18
101b3fc to
1ecdff0
Compare
tobiajo
force-pushed
the
tj/address-partition-ownership-overlap-possiblity-cassandra-persist-only
branch
from
July 11, 2026 14:05
1ecdff0 to
427f52b
Compare
tobiajo
changed the title
tobiajo
force-pushed
the
tj/address-partition-ownership-overlap-possiblity-cassandra-persist-only
branch
3 times, most recently
from
July 15, 2026 16:51
5922e19 to
06c302e
Compare
…st-only Snapshot persistence for the transactional Kafka mode was last-write-wins, so a stale writer (a zombie that lost its partition but is still flushing) could overwrite a newer snapshot. Guard the persist path with a compare-and-set on the offset: a write lands only if it advances the stored offset, so a stale writer's flush is rejected rather than clobbering the owner's state. This is the persist-only slice; delete stays plain last-write-wins. A guard-expired row (present with a null offset, left by a partial write whose TTL expired) is repaired by a persist claiming it via IF offset = null, which breaks the INSERT-IF-NOT-EXISTS conflict-forever deadlock. The design doc (docs/cassandra-single-writer-design.md, titled ": persist only") is rewritten around the choices and their learnings: why the offset, not the consumer generation, is the fence token, and where the subtlety lives. FlowSpec is adapted to the PartitionAssignment API. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PGxViqVaLYb1xQyveED6Hs
tobiajo
force-pushed
the
tj/address-partition-ownership-overlap-possiblity-cassandra-persist-only
branch
from
July 22, 2026 21:49
06c302e to
1a8c137
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on (merge first): #854
Problem
Consumer-group ownership of the input topic does not extend to the snapshot store (#732): a zombie that lost its partition but is still flushing can overwrite a newer snapshot, and the next recovery loads stale state while resuming from the newer committed offset. The Kafka backend closes this with transactional writes; the Cassandra backend was still last-write-wins.
Fix
Guard the persist path with a compare-and-set on the offset (a lightweight transaction): a write lands only if it advances the stored offset, so a stale writer's flush is rejected (
SnapshotWriteConflict) instead of clobbering the owner's state. The first write of a key usesINSERT ... IF NOT EXISTS, and a guard-expired row (present with a null offset after a partial write's TTL expiry) is repaired by claiming it viaIF offset = null, so no key can conflict forever. Opt-in:snapshotCompareAndSetonCassandraPersistence, off by default.This is the persist-only slice:
deletestays plain last-write-wins (offset-gating it is an API break, kept for the full compare-and-set mode stacked on this). Accepted residual: a stale writer can revive a just-deleted key. Nocorechange — buffer and recovery stay exactly as on master.Tests and docs
Store-level compare-and-set suite (
SnapshotSpec), TTL edge cases (SnapshotTtlEdgeSpec), and the flow-level corruption reproduction and prevention (FlowSpec). New design doc,docs/cassandra-single-writer-design.md— including why the offset, not the consumer generation, is the fence token here; persistence docs updated.