CrossProcessChangeFeedTest.AWriteInOneProcess_ReachesTheOtherProcessesLiveMirror_WithoutARecycle failed on #1988's shard 1 with the pre-fix signature from #1929 — the failure that fix exists to eliminate:
Expected value to be 2 because the mirror ADOPTS the durable version verbatim.
Against `main` this is where the test fails: the mirror ends at durable+1 — a revision
the store never held (#1432) — because the notification was dropped and the content only
arrived later, through a read-seeded mint. … but found 3.
It passed on re-run, and the six shards are now green.
Why this is not "just a flake"
The assertion is not a timing bound — it is a value comparison, and the value it found (durable+1 = 3) is specifically the shape #1432 describes: converging on content by accident, at a version that exists nowhere, because the notification was dropped and the content arrived later through a read-seeded mint.
So the observed behaviour is the notification losing a race with the read-seeded mint. #1929 closed that path in the steady case; intermittently it still loses. A green re-run tells you the race is not deterministic. It does not tell you it is gone.
What was ruled out
#1988 is not the cause. Its entire diff is one file — memex/Memex.Portal.Shared/MemexConfiguration.cs — adding a response-header middleware (app.Use → OnStarting → set headers) to the HTTP request pipeline. CrossProcessChangeFeedTest builds hubs against a shared store and runs no HTTP pipeline at all. It is rebuilt by the change (MeshWeaver.Hosting.Monolith.Test references Memex.Portal.Shared), so it is not out of reach by dependency — only by behaviour. The re-run passing is consistent with that.
What is NOT evidence, and should not be cited as such
The test passes locally on main — in 2 s for 3 tests, against bounds of [Fact(Timeout = 90_000)] and TestHardDeadline = 120 s. That is far too fast to have exercised the Postgres notify path CI uses, so the local run clears nothing. Anyone reproducing this needs to confirm which path they are actually on before drawing a conclusion.
Suggested next step
Establish the race rather than re-running until green: instrument the ordering between the delivered notification and the read-seeded mint, and assert the notification wins — or make the mint unable to advance the version when a notification for that revision is in flight. The current assertion is correct and should not be relaxed; it caught this.
Reported by me because the fix in #1929 is mine, so this is my debt rather than #1988's.
🤖 Generated with Claude Code
CrossProcessChangeFeedTest.AWriteInOneProcess_ReachesTheOtherProcessesLiveMirror_WithoutARecyclefailed on #1988's shard 1 with the pre-fix signature from #1929 — the failure that fix exists to eliminate:It passed on re-run, and the six shards are now green.
Why this is not "just a flake"
The assertion is not a timing bound — it is a value comparison, and the value it found (durable+1 = 3) is specifically the shape #1432 describes: converging on content by accident, at a version that exists nowhere, because the notification was dropped and the content arrived later through a read-seeded mint.
So the observed behaviour is the notification losing a race with the read-seeded mint. #1929 closed that path in the steady case; intermittently it still loses. A green re-run tells you the race is not deterministic. It does not tell you it is gone.
What was ruled out
#1988 is not the cause. Its entire diff is one file —
memex/Memex.Portal.Shared/MemexConfiguration.cs— adding a response-header middleware (app.Use→OnStarting→ set headers) to the HTTP request pipeline.CrossProcessChangeFeedTestbuilds hubs against a shared store and runs no HTTP pipeline at all. It is rebuilt by the change (MeshWeaver.Hosting.Monolith.TestreferencesMemex.Portal.Shared), so it is not out of reach by dependency — only by behaviour. The re-run passing is consistent with that.What is NOT evidence, and should not be cited as such
The test passes locally on
main— in 2 s for 3 tests, against bounds of[Fact(Timeout = 90_000)]andTestHardDeadline = 120 s. That is far too fast to have exercised the Postgres notify path CI uses, so the local run clears nothing. Anyone reproducing this needs to confirm which path they are actually on before drawing a conclusion.Suggested next step
Establish the race rather than re-running until green: instrument the ordering between the delivered notification and the read-seeded mint, and assert the notification wins — or make the mint unable to advance the version when a notification for that revision is in flight. The current assertion is correct and should not be relaxed; it caught this.
Reported by me because the fix in #1929 is mine, so this is my debt rather than #1988's.
🤖 Generated with Claude Code