fix(worker): pg_durable_worker hangs on graceful PostgreSQL shutdown - #321
Merged
Conversation
Root causes identified and fixed: 1. **Explicit duroxide pool close** - The duroxide store's PgPool (configured with min_connections=1) was only closed on Arc drop, not awaited. This left async cleanup tasks in the Tokio runtime that could block. Now explicitly closed with a 5s timeout in run_until_extension_dropped_or_shutdown. 2. **Bounded pool close timeouts** - mgmt_pool.close() and poll_pool.close() in run_duroxide_runtime now have 5s timeouts so a shutting-down PostgreSQL backend cannot block indefinitely. 3. **Reduced duroxide shutdown delay** - The 10s sleep in duroxide_runtime.shutdown() is reduced to 2s. Long-running work errors out anyway once PostgreSQL backends are gone; a 10s delay blocks the postmaster from completing its graceful stop within the pg_ctl timeout. 4. **Interruptible retry sleeps** - All tokio::time::sleep() calls in retry loops (pool creation, init retries, extension polling) are replaced with tokio::select! so shutdown is detected within 100ms instead of waiting the full sleep duration. 5. **shutdown_background() instead of shutdown_timeout(5s)** - All async cleanup is now done inside block_on(). Any remaining Tokio blocking-pool threads are killed by proc_exit() when the BGW exits anyway. 6. **pg-stop.sh / test-e2e-local.sh / test-upgrade.sh** - Stop functions now use -t 30 and fall back to immediate mode if fast stop times out, so test runs don't get stuck on a stale postmaster.pid. 7. **test-shutdown.sh** - New regression test script that exercises three scenarios (idle worker, active functions, rapid stop/start cycles) to verify the worker exits within a configurable timeout. Fixes #308 Co-authored-by: pinodeca <32303022+pinodeca@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Fix graceful PostgreSQL shutdown to avoid blocking pg_durable_worker
fix(worker): pg_durable_worker hangs on graceful PostgreSQL shutdown
Jul 30, 2026
pinodeca
marked this pull request as ready for review
July 30, 2026 22:18
- Add a shutdown regression step to the test job (--cycles 1, per PG matrix) - test-shutdown.sh: return instead of exit from start_server so Scenario C's restart-failure branch is reachable - test-shutdown.sh: surface build output on failure instead of swallowing it - test-shutdown.sh: clean up sed -i.bak backup files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pg_durable_workercould survivepg_ctl stop -m fast, keeping the postmaster alive until the stop timeout, blocking restarts and leaving a stalepostmaster.pid. The stuck process was observed infutex_wait_queue(Tokio blocking pool waiting on SQLx reconnection) anddo_epoll_wait(Tokio I/O reactor still polling dying connections).worker.rs — shutdown correctness
PgPool(min_connections=1) was never awaited on close — only dropped — leaving its maintenance task spinning. Now closed with a 5 s timeout after duroxide shutdown completes.mgmt_pool.close()andpoll_pool.close()are now wrapped intokio::time::timeout(5s)so a non-responsive backend can't block indefinitely.Runtime::shutdown(Some(10_000))unconditionally sleeps 10 s before aborting tasks. Reduced to 2 s — pending work errors out anyway once PG backends are gone.tokio::time::sleep()calls in retry/polling loops replaced withtokio::select!via a newwait_for_shutdown()helper, so shutdown is noticed within ~100 ms rather than waiting out the full sleep.rt.shutdown_timeout(5s)→rt.shutdown_background()— all async cleanup is now done insideblock_on; residual blocking threads are killed byproc_exit()on BGW exit.Scripts — stop resilience
pg-stop.sh,test-e2e-local.sh,test-upgrade.sh:stop -m fastnow uses-t 30; if it times out the scripts fall back to-m immediateautomatically instead of masking the failure with|| true.New:
scripts/test-shutdown.shRegression harness for this class of bug. Exercises three scenarios: worker idle at shutdown, worker mid-execution, and rapid stop/start cycles — each asserting the server stops within a configurable timeout (default 20 s).