test: fail the gate when a suite exists that no gate runs

[jdatcmd]

Four consecutive PRs added a suite without registering it in run_all_versions.sh, and each time I caught it by hand in review. Checking the whole tree found two more that had never been registered at all.

A suite no gate runs is not a test. It is a file that happens to contain assertions, and it stops being true the first time somebody changes the code under it.

What was actually unregistered

suite	how long
native_agg_deletes (#151)	caught in review
native_fetch_position (#152)	caught in review
analyze_stats (#159)	caught in review
write_minmax_fastpath (#160)	merged unregistered
native_reclaim_reconcile	never registered, predates all of this

write_minmax_fastpath is mine: I pushed the registration commit to the wrong branch name and #160 merged without it. Which is a fair illustration of why this should not depend on a reviewer noticing.

The check

harness_selftest now asserts both directions: every suite file appears in SUITES, and every name in SUITES has a file. The second catches a rename or a typo, which the runner would otherwise surface only when it tried to run the missing thing.

The allowlist is short and each entry carries its reason in the comment, because adding a name to it is the easy way to make this check go quiet: lib and run_all_versions are not suites, devloop and rebuild build rather than test, and native_scale is a suite but opt-in by design and says so in its own header, since it runs at a row count the matrix should not carry.

Proven by mutation, PG18

mutation	result
remove a suite from SUITES	FAIL unregistered: write_minmax_fastpath
add a SUITES name with no file	FAIL missing: native_typo_suite
add that suite to the allowlist instead	passes again, escape hatch behaving as documented
unmutated	both checks pass

The two newly registered suites pass

Both were run on PG18 before being added, so the matrix is not being handed a suite nobody has run: native_reclaim_reconcile exit 0, write_minmax_fastpath exit 0.

Gate

PG18 and PG19, ALL VERSIONS PASSED, 152 suite runs (75 per major, up from 73).

[ChronicallyJD]

Approving. Three of the five unregistered suites are mine, so this is a gap I created repeatedly and did not notice once.

Verified independently

I extracted the parser and ran it against the branch's own runner, because an awk/sed walk over a bash array is the fragile part of this and a parser that silently returns nothing would make the second check vacuous while the first got noisier:

names parsed from SUITES:  75      (matches the 75 per major in your gate line)

It also survives the two edges I tried to break it on — the first name on the SUITES=( line and the last before the closing paren both come through, which a naive grep -E '^[a-z0-9_]+$' without your sed drops. I confirmed that by running it without the sed and getting 73.

Registration checks out for native_agg_deletes, native_fetch_position, write_minmax_fastpath and native_reclaim_reconcile.

analyze_stats is correctly absent — it belongs to #159, which is not merged. I will register it in the same push as the fix, so it does not become the sixth entry in that table.

The allowlist is the right size

Each of the five has a reason that is about what the file is rather than about wanting the check quiet: lib and run_all_versions are not suites, devloop and rebuild build, and native_scale is opt-in with the row count as the stated reason. That last one is the only judgement call and I agree with it — a suite the matrix cannot afford is better excluded loudly, in a list with a comment, than included and then skipped by some flag.

The reverse check is the one I would have left out and shouldn't have. A SUITES entry with no file currently surfaces as a runner failure at the moment it tries to run it, which is late and reads like a broken test rather than a broken list.

On the mutation table

Proving the escape hatch works — allowlisting the missing suite makes it pass again — is worth as much as the two failures. It is the check most likely to be silenced under time pressure, and showing exactly how demonstrates that the only way to defeat it leaves a name in a list with a comment next to it, where review can see it.

What I take from this

My three were all the same mistake in the same place: I wrote the suite, ran it directly, saw it pass, and never asked what would run it tomorrow. Running a suite by hand proves it works once; registering it is what makes it a test. The check is the right fix precisely because it does not depend on a reviewer — you caught mine three times and the fourth still got through.

write_minmax_fastpath merging unregistered because a registration commit went to the wrong branch is the same class of accident and makes the argument better than a cleaner story would have.

Verdict

Approving. Correct, verified, and it closes a hole that four consecutive PRs walked through.