A blocking bridge in a test wedges the whole shard: methodTimeout cannot abort a thread parked in a native wait

[rbuergi]

Found while diagnosing #1991's shard-2 timeout, and generalised because the signature is greppable and has almost certainly bitten before.

The mechanism

A test that bridges an observable to a blocking call — .ToEnumerable(), Rx's IObservable<T>.Wait(), .Result, .GetAwaiter().GetResult() — parks the calling thread on a semaphore until the source produces. When the source schedules onto that same thread, it self-deadlocks.

Which thread it lands on depends on the scheduler the operator picks, so it is intermittent: in #1991 one full local run passed with the offending test present, and two later runs wedged for 31 and 15 minutes at ~0% CPU.

Why it costs a whole shard rather than one test

🚨 xUnit's methodTimeout cannot abort a thread parked in a native wait. So instead of a 30 s test failure you get an unbounded host wedge:

[CI] MeshWeaver.Graph.Test exit=124 TIMEOUT (8m wall-clock cap hit — likely fixture/init hang)

— exit=124, no failing test named, and the marker's own guess ("fixture/init hang") points away from the real cause. That is why the 30 s bound appeared to have been bypassed.

The stack that identifies it:

SemaphoreSlim.Wait()
System.Reactive…ObservableImpl.GetEnumerator.MoveNext()
System.Linq.Enumerable.ToArray
<the test>

Why this is not just "the codebase bans .Wait()"

It is the same rule AGENTS.md already states for product code — async IS a deadlock; compose with .Select/.SelectMany/.Subscribe — but tests have been treated as exempt, and the consequence there is worse, not milder: product code that deadlocks fails one request, while a test that deadlocks takes the shard's remaining tests with it and reports a cause that is not the cause.

Current exposure

• .ToEnumerable(): 0 remaining (the feat: presentation mode — a per-viewer, display-only privacy screen (#1803) #1991 instance is fixed on that branch).
• .Wait(): 37 call sites across 10+ test files — SystemScopeDoesNotEscapeTest, PostgreSqlTransientRetryTest, SyncedQueryPgTest, ContentChunkSearchTest, ChunkNavigationStoreTest, OverlayReEvaluationReadTest, NodeTypeBakeStatusTest, ThreadStreamingIdentityTest, …

Most are probably safe — e.g. pipeline.Wait().Should().Be(1) over an Observable.Return(1) completes before the wait blocks. This needs per-site judgement, not a blanket rewrite, which is exactly why it is filed rather than fixed.

What would close it

1. Triage the 37: a site is safe only if the source cannot schedule onto the calling thread. Where it can, subscribe-and-collect on ImmediateScheduler — the pattern feat: presentation mode — a per-viewer, display-only privacy screen (#1803) #1991 moved to, and the one the rest of that file already used.
2. Consider making the rule explicit for tests, so the next one is caught in review rather than as a wedged shard.
3. Cheapest and most valuable: treat any exit=124 with no named test as "a blocking bridge in a test" first. That signature is specific enough to grep for, and today it was misattributed twice — once to CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 (Roslyn emit poisoning) and once to machine contention, both plausible and both wrong.

Related: #890 (a genuinely different exit=124, distinguished by a Cci.MetadataWriter NRE and the emit canary), #612, #1843.

🤖 Generated with Claude Code