Skip to content

Clear the re-entrancy guard instead of restoring its previous value - #1047

Open
serioushaircut wants to merge 1 commit into
Shopify:mainfrom
serioushaircut:fix-adapter-reentrancy-guard-strand
Open

Clear the re-entrancy guard instead of restoring its previous value#1047
serioushaircut wants to merge 1 commit into
Shopify:mainfrom
serioushaircut:fix-adapter-reentrancy-guard-strand

Conversation

@serioushaircut

@serioushaircut serioushaircut commented Aug 31, 2026

Copy link
Copy Markdown

Fixes the one genuine race in #1045. The documentation half is #1046.

mark_resource_as_acquired captures @resource_acquired and restores it on the way out.

acquire_semian_resource is its only caller and returns early when the guard is already set, so in supported use — an adapter instance standing for one resource, used by one session at a time — the captured value can only ever be falsey!

Capturing it is what makes a violation of that contract permanent.

Problem recap - with the instance shared between threads:

  1. B reads the guard as false and proceeds.
  2. A sets it: previous = false, guard = true.
  3. B sets it: previous = true, guard = true.
  4. A finishes: guard = false.
  5. B finishes: guard = true, and nothing ever clears it. ❗

From then on every call through that instance returns early from acquire_semian_resource:

  • no fast-fail on an open circuit
  • no failure accounting
  • no :success or :circuit_open notifications
    while the circuit still reports closed.

The window between 1 and 2 is not theoretical. ProtectedResource#acquire notifies :success before yielding, so any subscriber that writes to a socket — a StatsD backend, say — releases the GVL inside it. With one such subscriber the strand reproduced in 75 of 75 rounds of 8 threads × 200 calls; with no subscriber, not once in ~10M calls.

Assigning false in the ensure is the value previous always holds in supported use, so nothing changes there, and a shared instance degrades transiently instead of permanently.

Tests. The guard had none. This adds the interleaving above — deterministic, driven
with queues rather than sleeps — and one asserting that nesting still short-circuits, so
a nested call neither re-enters the circuit breaker nor takes a second bulkhead ticket.

Alternative, if you would rather the racy case were handled fully: a depth counter
under a per-instance mutex, incremented and decremented rather than saved and restored,
keeps nesting protection intact even when the instance is shared. It costs one uncontended
mutex per acquire and needs double-checked lazy init for the mutex. Happy to send that
version instead.

Verified locally against 0.28.2 on CRuby 3.4.10: the new test fails before the change and
passes after, and the reproduction from #1045 goes from a permanently dead breaker to one
that keeps opening.

Co-authored-by: Claude Opus 5 noreply@anthropic.com
Orchestrated-by: ae noreply@shopify.com

acquire_semian_resource is the only caller of mark_resource_as_acquired
and it returns early when the guard is already set, so in supported use
the value captured as `previous` can only ever be falsey. Capturing and
restoring it is what turns a violation of that contract into a permanent
one: if two callers interleave, one captures true, the other restores
false, and the one holding true restores it after both have left. From
then on every call on that instance returns early from
acquire_semian_resource, so the circuit breaker is disabled for the
lifetime of the process, silently and with the circuit still reporting
closed.

Assign false on the way out instead. In supported use this is the value
`previous` always held, so behaviour is unchanged; a shared instance now
degrades transiently rather than permanently.

Two tests, since the guard had none: the interleaving above, and nesting
still short-circuiting so a nested call neither re-enters the circuit
breaker nor takes a second bulkhead ticket.

Refs Shopify#1045

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Orchestrated-by: ae <noreply@shopify.com>
Assisted-By: devx/39c7b308-c333-4ffe-8cd7-f642a14a4b7c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants