Drop client queries abandoned during maintenance mode#1244
Open
crodas wants to merge 2 commits into
Open
Conversation
Maintenance mode parked a fully-buffered request on a bare await that ignored the client socket. If the client gave up and disconnected while the database was paused, its query was still dispatched to the backend once maintenance lifted, running work for a client that was already gone. Wait for maintenance to lift while watching the client socket, so a disconnect aborts the request before it reaches the query engine. Add a cancel-safe MessageBuffer::read_more that detects the disconnect without consuming any pipelined data, and cover the behavior with an integration test that asserts a write from a disconnected client never lands.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Collaborator
|
I think a simpler solution could just be to put the maintenance check at the top of |
The disconnect fix for maintenance mode added a separate wait_for_maintenance method, a small result enum, and matching branches in the client run loop. buffer() already owns the client socket and already turns an EOF into a disconnect, so it is the natural home for the wait and the extra scaffolding is redundant. Fold the gate into buffer(): once a request is fully buffered, park on the maintenance waiter while watching the socket, and on a disconnect clear the request and return the existing abrupt-disconnect event. This removes wait_for_maintenance, its result enum, and the run-loop plumbing. The check goes at the bottom of buffer, after the read loop, not at the top. A client sits idle inside buffer's read loop between requests; if maintenance turns on while it is parked there, a top-of-function check has already run and would let the next query through. Gating right before the buffered request is returned keeps the hold-at-dispatch behavior and is exercised by the existing idle-client maintenance test. The gate no longer sets the frontend Waiting state during a park, because buffer cannot borrow the query engine: the run loop already holds it to read the backend. A parked client keeps its prior state in SHOW CLIENTS, an accepted trade for the simpler shape.
Contributor
Author
|
I think 21764a1 is simpler |
meskill
reviewed
Jul 21, 2026
|
|
||
| #[tokio::test] | ||
| #[serial] | ||
| async fn test_maintenance_mode_drops_disconnected_client_query() { |
Contributor
There was a problem hiding this comment.
this test succeeds on main without fix.
I think it's because the client.execute will execute parse/describe/sync first and this blocked by maintenance before next execution round-trip anyway. So, try simple_query I guess
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attempt to fix #1242
Maintenance mode parked a fully-buffered request on a bare await that ignored the client socket. If the client gave up and disconnected while the database was paused, its query was still dispatched to the backend once maintenance lifted, running work for a client that was already gone.
Wait for maintenance to lift while watching the client socket, so a disconnect aborts the request before it reaches the query engine. Add a cancel-safe MessageBuffer::read_more that detects the disconnect without consuming any pipelined data, and cover the behavior with an integration test that asserts a write from a disconnected client never lands.