Skip to content

usenet-drain-walk: a busy host is not a stuck drain - #122

Merged
leonardoazeredo merged 13 commits into
mainfrom
fix/drain-walk-gate-refusals
Sep 22, 2026
Merged

leonardoazeredo merged 13 commits into
mainfrom
fix/drain-walk-gate-refusals

Conversation

@leonardoazeredo

Copy link
Copy Markdown
Owner

What was wrong

A real 12-pass run of scripts/usenet-drain-walk.sh on 2026-09-22 logged 4 of its 12 passes as "no progress" — and every one of those four was a pass the I/O pressure gate refused before it started. Not one of them ran. Two more refusals and the walk would have stopped printing 4 passes in a row made no progress: a sentence about a drain that stopped moving, printed for passes that never happened, and the first line an operator reads.

The gate prints [pressure-gate] on four paths and refuses on one. The other three say this pass runs unprotected and then run the pass normally — so the walk's detection (a bare prefix match) classified a pass that ran as one that never started.

The change

  • The refusal is its own counter. REFUSED (consecutive, reset by any admitted pass) no longer writes BARREN. A pass the gate refused says nothing about whether the drain is moving.
  • It has its own bound and reason. --max-skipped (default 6 — six skip-cooldowns, half an hour of a host too busy to start a pass) stops the walk saying the I/O pressure gate refused N passes in a row (the host was too busy to start one), instead of borrowing the barren sentence.
  • The summary counts them. passes run: 12 (4 refused by the pressure gate, 0 in a row with no progress at the end) — the line that would have made that run readable without cross-referencing the pass log by hand.
  • The detection matches the refusal's own phrase, and a test drives the real usenet-blackhole.sh through both gate states (refused, and "runs unprotected"), because a stub hardcodes the phrase and cannot keep it honest.
  • The final refusal no longer sleeps. It logged waiting 300s before the next attempt and slept it, although the bound guarantees no next attempt.
  • Also fixes a shipped --help truncation found on the way: the range stopped three lines short of the header, so the operator's own --help was missing the Prerequisites line and the warning beneath it. A test now derives the last header line from the file, so it cannot fall behind again.

Verification

  • The walk's own suite: 25 tests, all green. The mutation corpus reports killed 8 / 8, survived 0 — five new entries registering the guards this branch adds, each with a named test that goes red under it.
  • On the NAS at this HEAD, with the gate forced closed by PSI_IO_LIMIT=0 (no TorBox call, no download): the new refusal lines, the new stop reason, passes run: 2 (2 refused by the pressure gate, 0 in a row with no progress at the end), exit 3, and 2 refusals with 1 wait — the final one going straight to the stop. --help prints the whole header; usenet-blackhole.timer is still inactive.
  • Local full-suite caveat, stated rather than buried: ./tests/run-tests.sh reports executed=1112 failed=28 on this macOS host. Running the same six files on main fails the identical 28 test names — zero introduced, zero masked. They are host artifacts (ssh blocked by the sandbox, no docker compose CLI, no PyYAML, BSD sed, not on VLAN20). CI is the suite run to judge.

Notes

  • CLAUDE.md's rule that every code change is NAS-tested before main is met: synced and verified at 0250a1b, and the NAS goes back to main when this lands.
  • The plan (docs/superpowers/plans/2026-09-22-drain-walk-gate-refusals.md) travelled with the work, including three in-place corrections made when its own reviews disproved what it prescribed.
  • One known gap, ledgered: the streak-reset guard (REFUSED=0) has no test that fails if it is removed, so its killability is unproven. The shipped behaviour is correct; the evidence is not there yet.

Separating a pass the I/O pressure gate refused from a pass that ran and moved
nothing, so a saturated host stops the walk on its own reason instead of
reporting a stuck drain. Measured 2026-09-22: 4 of 12 passes were refusals and
all four were counted as "no progress", against a run in which no admitted pass
failed to move anything.
The pressure gate refusing a pass is the host protecting itself, and the pass
never starts -- scripts/usenet-blackhole.sh exits at the gate before python. It
was counted as 'no progress' anyway, so four refusals in one 12-pass run read
as a drain that had stopped moving. Measured 2026-09-22: 4 of 12 passes refused,
every admitted pass moved something.
The plan said a refused pass was one whose output contains [pressure-gate].
That prefix is printed on four paths in scripts/usenet-blackhole.sh and only
one refuses; the other three say the pass runs unprotected and then run it.
Recorded in place, with what the review found, because a future reader follows
this document.
The range stopped three lines short, so the operator's --help was missing the
Prerequisites line and the warning beneath it. The test that was supposed to
notice only named strings from the middle of the block. It now derives the last
header line from the file and asserts it appears, so the range cannot fall
behind the header again without going red.
Six refusals in a row now end the walk saying the I/O pressure gate refused
every pass, instead of '4 passes in a row made no progress' -- a sentence about
a drain that stopped moving, printed for passes that never started. --max-skipped
makes the bound a flag because the value is a guess: 6 skip-cooldowns is half an
hour of a host too busy to start a pass, against a measured worst streak of 2.
The plan prescribed a closing note claiming no pass ran at all, which the stop
condition does not guarantee: MAX_SKIPPED counts consecutive refusals and an
admitted pass resets the streak, so a run with earlier passes reaches that stop
too. Recorded in place with what the review found.
passes run: 12 (4 refused by the pressure gate, 0 in a row with no progress at
the end) is the line that would have made the 2026-09-22 run readable without
cross-referencing the pass log by hand.
The mutation puts the increment back on BARREN, which is the branch this work
removed, and requires the new test to go red. The same commit closes the
coverage gap Task 4's review left: nothing pinned REFUSED_TOTAL's
never-reset property.
Four of the guards this branch adds had no corpus entry: the narrowed refusal
match, the refusal bound, the cumulative refusal counter, and the --max-skipped
minimum. Each entry is verified by the harness -- 7 killed / 7, survived 0,
errored 0, skipped 0 -- and the first names the test that drives the real
scripts/usenet-blackhole.sh, because the refusal phrase is that script's own
output and a stub hardcodes it.
The last refusal took REFUSED to --max-skipped and then still logged and slept
the 300s skip-cooldown, although the loop top stops the walk before another
pass could run: five minutes of silence at the end of the one run an operator
watches, read as a hang. The test now asserts exactly one wait for two
refusals, which was verified to go red when the guard is removed.

--help's WHAT IT DOES said the walk stops after --max-barren but not after
--max-skipped, so the range was re-derived for the two lines that adds (86 to
88). The mixed-state test now checks its own premise (three passes: one
admitted, two refused), and the hardcoded-range comment in the derived-range
test no longer names a number that is now the shipped one.

The plan still prescribed PASS_SKIPPED and the prefix grep, so its code blocks
show PASS_REFUSED and the skipping-this-pass match, its test totals read 25,
and Task 6 Step 5 expects what a dry run actually prints -- the bounds are
shown under --apply only.
The wait guard this wave added to the cooldown block is a new guard by the same
rule as the other five, so it gets the corpus entry the first pass left out.
The entry drops the guard and names the test whose wait-line count catches it:
killed 8 / 8, survived 0, errored 0, skipped 0.

The plan's Task 5 Step 2 total follows it to eight.
@leonardoazeredo
leonardoazeredo merged commit cbee1aa into main Sep 22, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant