usenet-drain-walk: run the drain down one pass at a time, under a watchdog

[leonardoazeredo]

Why

The drain has been held down since 2026-09-20 (documented in docs/MAINTENANCE.md, "Holding the usenet ingest down"), and the sanctioned way to work it off is hand-running passes and reading each one. At 607 NZBs that stops being practical, and the alternative — arming the timer — is what wedged the box in the first place.

This is that manual procedure with the reading done by a rule.

What it does

• One bounded pass at a time, in the foreground, at the unit's own settings (--report-failures --max-inflight 6).
• A progress fingerprint — outbox depth, watch folder contents, staging bytes, state file size and mtime — sampled every 30s. The pass's whole process group is killed when none of it has moved for ten minutes.
• After each pass, whether the drain got anywhere at all. Four passes in a row that did not ends the walk.
• Exits 0 when the outbox ends below the mark, 3 when it does not, 1 for a refused precondition, 2 for a bad argument.

Killing a pass moves the walk on rather than repeating it, and nothing has to remember which release was stuck: run() marks last_fetch_attempt before it fetches and the fetch set is least-recently-attempted first, so a killed pass sends its releases to the back. That holds across a SIGKILL, not just a SIGTERM.

What it deliberately never does

• Never writes logs/usenet-blackhole-state.json — the Python half's file, on a resume contract, and a second writer on it is the coupling trap the pressure gate avoids on purpose.
• Never deletes an NZB — the outbox is the arrs' work queue.
• Never arms, stops or disables usenet-blackhole.timer. It refuses to run while that timer is active instead, because two drains on one outbox is the load this exists to avoid.

Verification

• 16 bats tests (tests/usenet-drain-walk.bats), all passing locally.
• Mutation corpus entry: the suite is killed by a slack outbox comparison and by a kill that reaches the shell but not the fetch underneath it.
• Dry run on the NAS against the live tree, resolving the same paths usenet-blackhole.sh prints (a test asserts the two agree, because a driver watching a different folder reads every running pass as stalled):

Mark: outbox below 50 NZBs
  NZB folder:   /volume1/data/usenet/blackhole/nzb
  watch folder: /volume1/data/usenet/blackhole/complete
  staging:      /volume1/data/usenet/blackhole/staging
Outbox now:   607 NZBs
Drain now:    jobs/complete/stalled/outbox = 22 21 1|607

21 of those 22 jobs are complete at TorBox and owed a local fetch, ~99 hours old: nothing has drained since 2026-09-20 21:25.