Skip to content

fix(worker): pg_durable_worker hangs on graceful PostgreSQL shutdown - #321

Merged
pinodeca merged 3 commits into
mainfrom
copilot/fix-pg-durable-worker-hang
Jul 31, 2026
Merged

fix(worker): pg_durable_worker hangs on graceful PostgreSQL shutdown#321
pinodeca merged 3 commits into
mainfrom
copilot/fix-pg-durable-worker-hang

Conversation

Copilot AI commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

pg_durable_worker could survive pg_ctl stop -m fast, keeping the postmaster alive until the stop timeout, blocking restarts and leaving a stale postmaster.pid. The stuck process was observed in futex_wait_queue (Tokio blocking pool waiting on SQLx reconnection) and do_epoll_wait (Tokio I/O reactor still polling dying connections).

worker.rs — shutdown correctness

  • Explicit pool close: The duroxide store's PgPool (min_connections=1) was never awaited on close — only dropped — leaving its maintenance task spinning. Now closed with a 5 s timeout after duroxide shutdown completes.
  • Bounded pool close: mgmt_pool.close() and poll_pool.close() are now wrapped in tokio::time::timeout(5s) so a non-responsive backend can't block indefinitely.
  • Duroxide shutdown delay: Runtime::shutdown(Some(10_000)) unconditionally sleeps 10 s before aborting tasks. Reduced to 2 s — pending work errors out anyway once PG backends are gone.
  • Interruptible retry sleeps: All bare tokio::time::sleep() calls in retry/polling loops replaced with tokio::select! via a new wait_for_shutdown() helper, so shutdown is noticed within ~100 ms rather than waiting out the full sleep.
  • Tokio runtime shutdown: Changed rt.shutdown_timeout(5s)rt.shutdown_background() — all async cleanup is now done inside block_on; residual blocking threads are killed by proc_exit() on BGW exit.
// Before: pool just dropped, maintenance task keeps running
drop(duroxide_runtime);

// After: explicit close with timeout
let _ = tokio::time::timeout(
    Duration::from_secs(5),
    duroxide_store.pool().close(),
).await;

Scripts — stop resilience

  • pg-stop.sh, test-e2e-local.sh, test-upgrade.sh: stop -m fast now uses -t 30; if it times out the scripts fall back to -m immediate automatically instead of masking the failure with || true.

New: scripts/test-shutdown.sh

Regression harness for this class of bug. Exercises three scenarios: worker idle at shutdown, worker mid-execution, and rapid stop/start cycles — each asserting the server stops within a configurable timeout (default 20 s).

Root causes identified and fixed:

1. **Explicit duroxide pool close** - The duroxide store's PgPool (configured
   with min_connections=1) was only closed on Arc drop, not awaited. This left
   async cleanup tasks in the Tokio runtime that could block. Now explicitly
   closed with a 5s timeout in run_until_extension_dropped_or_shutdown.

2. **Bounded pool close timeouts** - mgmt_pool.close() and poll_pool.close()
   in run_duroxide_runtime now have 5s timeouts so a shutting-down PostgreSQL
   backend cannot block indefinitely.

3. **Reduced duroxide shutdown delay** - The 10s sleep in
   duroxide_runtime.shutdown() is reduced to 2s. Long-running work errors out
   anyway once PostgreSQL backends are gone; a 10s delay blocks the postmaster
   from completing its graceful stop within the pg_ctl timeout.

4. **Interruptible retry sleeps** - All tokio::time::sleep() calls in retry
   loops (pool creation, init retries, extension polling) are replaced with
   tokio::select! so shutdown is detected within 100ms instead of waiting
   the full sleep duration.

5. **shutdown_background() instead of shutdown_timeout(5s)** - All async
   cleanup is now done inside block_on(). Any remaining Tokio blocking-pool
   threads are killed by proc_exit() when the BGW exits anyway.

6. **pg-stop.sh / test-e2e-local.sh / test-upgrade.sh** - Stop functions now
   use -t 30 and fall back to immediate mode if fast stop times out, so test
   runs don't get stuck on a stale postmaster.pid.

7. **test-shutdown.sh** - New regression test script that exercises three
   scenarios (idle worker, active functions, rapid stop/start cycles) to
   verify the worker exits within a configurable timeout.

Fixes #308

Co-authored-by: pinodeca <32303022+pinodeca@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix graceful PostgreSQL shutdown to avoid blocking pg_durable_worker fix(worker): pg_durable_worker hangs on graceful PostgreSQL shutdown Jul 30, 2026
Copilot AI requested a review from pinodeca July 30, 2026 21:00
@pinodeca
pinodeca marked this pull request as ready for review July 30, 2026 22:18
- Add a shutdown regression step to the test job (--cycles 1, per PG matrix)
- test-shutdown.sh: return instead of exit from start_server so Scenario C's
  restart-failure branch is reachable
- test-shutdown.sh: surface build output on failure instead of swallowing it
- test-shutdown.sh: clean up sed -i.bak backup files
@pinodeca
pinodeca merged commit b0cd9ed into main Jul 31, 2026
5 checks passed
@pinodeca
pinodeca deleted the copilot/fix-pg-durable-worker-hang branch July 31, 2026 01:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Graceful PostgreSQL shutdown can hang pg_durable_worker and block restart

2 participants