fix streaming stuck after snapshot - #70
Merged
Merged
Conversation
hasyimibhar
added a commit
that referenced
this pull request
Jun 3, 2026
A second reader/main-loop deadlock (the first was #70) could still freeze CDC after the snapshot under sustained write load. The reader task pushes decoded events into a bounded channel and, in the same task, services `Cmd::Standby` slot-ack requests. When the events channel filled, the reader parked inside a bare `events_tx.send(msg).await` — which is *outside* its `select!` — so it stopped polling `cmd_rx`. Meanwhile the main loop's Standby tick blocks in `send_standby` awaiting the reader's ack and, while blocked there, stops draining the events channel. Neither side can progress: the reader waits for a free slot the main loop will never make, and the main loop waits for an ack the reader can never send. All tokio workers park; the slot lag grows without bound. #70 biased the reader's `select!` to `cmd_rx` first, but that doesn't help here: the reader never reaches the `select!` because it's blocked in the channel send before it. Fix: route every event send through `send_servicing_cmds`, which `select!`s the channel send permit (`reserve()`) against `cmd_rx.recv()`. A `Standby` is now acked even when the events channel is full, so the main loop resumes, drains the channel, frees a slot, and the send completes. `Cmd` handling is factored into `handle_cmd`, shared by the main select arm and the send-wait path. Reproduced on example/single (iceberg-rest + minio, no AWS) under 8 concurrent writers — froze within seconds before, runs indefinitely after (slot lag drains to ~0 each flush cycle). Regression test: `reader_tests::standby_acked_while_events_channel_full`. Found via the AWS benchmark; localized with a tokio task dump.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Another issue introduced by Rust port.