Heal the auth mirror in batches the caller drives, not one blocking sweep by rbuergi · Pull Request #2061 · Systemorph/MeshWeaver

rbuergi · 2026-08-22T10:08:01Z

You were right that raising the timeout was the wrong shape. This is the fix; #2058 is the floor.

What was wrong

The self-heal swept every partition schema inside a single DO block that the client waited on while holding pg_advisory_xact_lock for the whole sweep. Three problems, and only the first is a timeout:

the client blocked on one statement whose duration scales with the size of the mesh — a big instance hit Npgsql's default and the migration CrashLoopBackOff'd, while the same image migrated a smaller instance fine (memex.meshweaver.cloud, 2026-08-22);
a timeout abandons the client while the server keeps going — the work is neither finished nor cancelled, and nothing knows which;
the global lock was held for the entire sweep, so every concurrently-booting silo queued behind it. The contention grows with the very thing that made it slow.

Raising the timeout answers only (1), with a number that has to keep growing as the mesh does.

Trigger, then drive to completion

public.mw_auth_mirror_heal_batch(p_after text, p_limit int) RETURNS text

Heals at most p_limit schemas after the cursor and returns the last one it touched; the caller loops until it returns NULL. Each call takes the lock, commits, and releases it.

no statement is long, so no timeout scales with the mesh
another silo can interleave instead of queueing behind the whole sweep
a dropped connection costs one batch, not the sweep — the next call resumes at the cursor, because the work was always idempotent per schema

🚨 GetAuthMirrorSelfHealScript() is no longer "the heal" — it only defines the function. RunAuthMirrorSelfHealAsync is the operation, so no caller can half-run it by executing the script alone.

Verified against a real Postgres

Testcontainers, not shape-tests: 15 heal tests green across AuthMirrorTriggerTests, AccessTriggerSchemaResolutionTests and GroupMembershipRecomputeTests. Whole assembly 739/741 — the single failure is UserDirectoryCompletenessTests, which is #2031 and fails on main too.

Two mistakes the tests caught, worth naming because both are invisible to a compiler:

a bare RETURN; is valid in a DO block and a syntax error in a function returning text (42601 — surfaced as the whole fixture failing to initialize)
defining the function is not running it: three tests executed the script directly and silently healed nothing

Relationship to #2058

#2058 raises the blanket init timeout to 600s. Keep both: this removes the need for a large number, #2058 remains a sane floor for the other boot-time DDL. If #2058 lands first this needs a trivial rebase.

…weep The self-heal swept every partition schema inside a single DO block that the client waited on while holding pg_advisory_xact_lock for the WHOLE sweep. Three things were wrong with that, and only the first is a timeout: 1. the client blocked on one statement whose duration scales with the size of the mesh, so a big instance hit Npgsql's default and the migration CrashLoopBackOff'd — the same image migrating a smaller instance fine (memex.meshweaver.cloud, 2026-08-22); 2. a timeout ABANDONS the client while the server keeps going: the work is neither finished nor cancelled, and nothing knows which; 3. the global lock was held for the entire sweep, so every concurrently booting silo queued behind it — the contention grows with the very thing that made it slow. Raising the timeout answers only (1), and answers it with a number that has to keep growing as the mesh does. So the work is TRIGGERED in bounded batches and driven to completion instead: public.mw_auth_mirror_heal_batch(p_after text, p_limit int) RETURNS text heals at most p_limit schemas after the cursor and returns the last one it touched; the caller loops until it returns NULL. Each call takes the lock, commits, and releases it. No statement is long, another silo can interleave, and a dropped connection costs one batch instead of the sweep — the next call resumes at the cursor, because the work was always idempotent per schema. 🚨 GetAuthMirrorSelfHealScript() is no longer "the heal" — it only DEFINES the function. RunAuthMirrorSelfHealAsync is the operation, so no caller can half-run it by executing the script alone. The three tests that executed the script directly now call it; that is exactly how this was caught. Verified against a real Postgres (Testcontainers): 15 heal tests green across AuthMirrorTriggerTests, AccessTriggerSchemaResolutionTests and GroupMembershipRecomputeTests; 739/741 for the whole assembly, the one failure being UserDirectoryCompletenessTests, which is #2031 and fails on main. Two mistakes the tests caught, worth naming: a bare `RETURN;` is valid in a DO block and a syntax error in a function returning text (42601, surfacing as the fixture failing to initialize); and defining the function is not running it.

Copilot

Pull request overview

This PR replaces the unbounded auth-mirror sweep with caller-driven batches and updates integration tests to invoke the complete healing operation.

Changes:

Adds bounded PostgreSQL heal batches and cursor-driven execution.
Applies per-batch command timeouts.
Updates auth, access-trigger, and group-membership tests.

Outstanding review items include lock-safe global DDL setup, cursor recovery after failures, 26-schema batch-boundary coverage, and stale XML documentation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
`test/MeshWeaver.Hosting.PostgreSql.Test/GroupMembershipRecomputeTests.cs`	Updates group-membership healing coverage.
`test/MeshWeaver.Hosting.PostgreSql.Test/AuthMirrorTriggerTests.cs`	Tests auth-mirror reconciliation through the new operation.
`test/MeshWeaver.Hosting.PostgreSql.Test/AccessTriggerSchemaResolutionTests.cs`	Updates access projection healing coverage.
`src/MeshWeaver.Hosting.PostgreSql/PostgreSqlSchemaInitializer.cs`	Implements batched self-healing and cursor iteration.

Suppressed comments (3)

src/MeshWeaver.Hosting.PostgreSql/PostgreSqlSchemaInitializer.cs:359

This installs the new batch function before any auth-heal advisory lock is acquired. InitializeAsync is invoked by each booting silo, so concurrent CREATE OR REPLACE FUNCTION statements can race on the same pg_proc row and raise the tuple concurrently updated failure described by AcquireSchemaInitLockAsync below. The per-batch lock cannot protect this command; serialize the install under the same cross-silo lock before driving the batches.

        await using (var install = dataSource.CreateCommand(GetAuthMirrorSelfHealScript()))
        {
            // Installing the function is plain DDL — short by construction.
            install.CommandTimeout = BatchCommandTimeoutSeconds;
            await install.ExecuteNonQueryAsync(ct).ConfigureAwait(false);

src/MeshWeaver.Hosting.PostgreSql/PostgreSqlSchemaInitializer.cs:666

The 25-schema limit does not bound this batch when it repairs a Group or GroupMembership row: the upsert into auth.mesh_nodes later in the same function fires zzz_group_recompute_*, whose GroupChangedTriggerFunctionBody loops every partition with a group grant and performs a full rebuild. A batch containing stale/missing group rows can therefore still fan out across the whole mesh (and repeat once per row), defeating the timeout/scaling fix. Suppress or defer that trigger during mirror backfill, then process the affected projection work in bounded units.

                  AND t.table_schema NOT IN
                      ('information_schema','pg_catalog','pg_toast','public','admin','auth')
                  AND t.table_schema NOT LIKE '%\_versions'
                  AND (p_after IS NULL OR t.table_schema > p_after)
                ORDER BY t.table_schema
                LIMIT p_limit

src/MeshWeaver.Hosting.PostgreSql/PostgreSqlSchemaInitializer.cs:312

This is a user-noticeable production fix: it prevents large deployments from failing migration and entering CrashLoopBackOff, so it is not an internal-only change. Add the required per-PR src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-22-<slug>.md entry with Category: Fix, a description, icon, and negative date order.

            await RunAuthMirrorSelfHealAsync(dataSource, ct).ConfigureAwait(false);

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+            -- BOUNDED, ORDERED slice after the caller's cursor. The caller keeps calling with the
+            -- last name returned until this returns NULL, so no single statement is long and the
+            -- advisory lock above is held for ONE BATCH rather than the whole mesh.


+        string? after = null;
+        while (true)
+        {
+            await using var batch = dataSource.CreateCommand(
+                "SELECT public.mw_auth_mirror_heal_batch($1, $2)");


+    private const int AuthMirrorHealBatchSize = 25;
+
+    /// <summary>Per-batch ceiling. Bounded work deserves a bounded, modest timeout — the point of
+    /// batching is that no single statement scales with the size of the mesh.</summary>
+    private const int BatchCommandTimeoutSeconds = 120;


+        CREATE OR REPLACE FUNCTION public.mw_auth_mirror_heal_batch(p_after text, p_limit int)
+        RETURNS text AS $auth_mirror_heal$


github-actions · 2026-08-22T10:21:21Z