Skip to content

Bound every CI job with a timeout, and install the audit tools - #2

Open
tony wants to merge 2 commits into
mainfrom
github-actions-savings
Open

tony wants to merge 2 commits into
mainfrom
github-actions-savings

Conversation

@tony

@tony tony commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Three commits: a timeout on every job, plus the two CI fixes this branch needs to be able to prove itself.

Bound every job with a timeout

GitHub's default job cap is six hours. Nothing here bounds a job that stops making progress, and one already has: reusable-deploy.yml's publish ran 81.6 minutes and then failed in Run aws-actions/configure-aws-credentials@v6 (run 34052068554), against a 1.24-minute median across its other September runs. publish-root shows a milder 11-minute instance against a 0.4-minute median.

This does not make CI faster and is not meant to. timeout-minutes is inert until a job overruns; measured across this series, job execution time is unchanged. What improves is time-to-signal on the pathological path: a hung publish now fails in ten minutes instead of eighty-one, and the six-hour ceiling stops being reachable at all.

Values are derived per job rather than chosen — 2.5x p95 or 1.2x the observed maximum, whichever is larger, rounded up to five minutes with a ten-minute floor. Observations above five times the median are dropped as hangs, unless three or more sibling jobs in the same run are also slow, which is a cold cache rather than a hang.

Workflow Job Median p95 Timeout
deploy-shell.yml locales 0.10 0.12 10
deploy-shell.yml build 0.75 2.51 10
deploy-shell.yml publish-root 0.68 1.59 10
pr-preview-cleanup.yml cleanup 15
reusable-deploy.yml publish 1.24 3.02 10
test.yml test 0.58 0.60 10

cleanup never fired in August or September, so its fifteen minutes is a default rather than a measurement — worth a second look.

Getting this branch to a state where it can prove itself

test.yml has never passed on main — all fourteen runs failed. It is broken in three independent ways, and this branch fixes the first two by cherry-picking the exact commits from trim-docs-wording (#1), so whichever lands first the other merges without conflict.

ba976d3 and e04d323 (from 4be0786 and 99e30e8) install the audit tools. Four scripts shell out to a developer toolbelt the ubuntu-latest image does not ship: scripts/check-palette.mjs, scripts/check-xrefs.mjs and scripts/normalise-parity.mjs call rg, and scripts/gen-mcp-tools.mjs calls fd. Without them the job died with spawnSync rg ENOENT before reaching anything this branch touches.

5cf2f77 (from 0f9d218) makes the example-sources negative test build its own fixture checkouts in a temp directory and point the generator at them through LIBTMUX_DOCS_CHECKOUT_<PORT>. Previously it depended on sibling port clones existing on the machine, so on a runner the generator found nothing, the staleness check passed vacuously, and the negative test correctly reported that the gate could not fail. Verified locally: all three cases now pass.

The third failure is not fixed here and should not be. The link check reports one broken target, /en/py/latest/api/, linked 58 times from en/py/index.html. That route comes from the site rewrite on #1, which touches 33 files under site/src/ including reference/[...slug].astro. Fixing it here would mean absorbing most of a 63-commit branch into a CI-bounds change.

So this pull request still shows test red. That is pre-existing on main, it is two layers better than it was, and it goes green when #1 merges. What is deliberately not carried over is #1's switch from pnpm test to pnpm test:publication — on that branch test:publication is the same ./scripts/test-all.sh this branch already runs, so the rename would change nothing here while conflicting with #1.

Unchanged on purpose

Concurrency is already correct in this repository. deploy-shell.yml and pr-preview-cleanup.yml use queue: max, which serialises deploys rather than killing them mid-publish, and that is the right choice for a deploy.

The credential hang itself is also untouched. A timeout caps the bleeding at ten minutes; it does not explain why the OIDC exchange hangs rather than failing fast. The failing run was a same-repo pull request, so a token was available to it — which points at the trust policy rather than at a missing token.

@tony

tony commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

The failing test check is pre-existing and unrelated to this PR.

scripts/check-palette.mjs:35 calls execFileSync('rg', ...), and ripgrep is not installed on ubuntu-latesttest.yml has no install step for it. The job dies with spawnSync rg ENOENT before reaching anything this branch touches.

The last eight test runs on main all failed the same way: 34032947807, 34032597272, 34030547283, 34029719292, 34029304389, 34028534729, 34025154452, 34024937643. This gate has not passed since the repo was created.

Two ways out, whichever you prefer: add sudo apt-get install -y ripgrep to test.yml, or drop the rg dependency from check-palette.mjs and walk the files in Node. Happy to open that as a separate PR — it is not in scope here.

@tony

tony commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Correction to my previous comment: the fix already exists, so there is nothing to open separately.

trim-docs-wording (#1) carries it in two commits — 4be0786 "fix(ci) Install ripgrep for publication audits" and 99e30e8 "fix(ci) Install fd for documentation audits" — which add an Install audit tools step to test.yml for ripgrep and fd-find, and switch the run to pnpm test:publication. Neither commit is on main.

That is why every test run on main has failed and the passing runs are all on trim-docs-wording. This branch is cut from origin/main, so it inherits the breakage rather than causing it.

I am deliberately not duplicating the install step here — it would conflict with #1. This check goes green once #1 merges.

For the record, four scripts depend on a developer toolbelt the ubuntu-latest image does not ship: scripts/check-palette.mjs, scripts/check-xrefs.mjs and scripts/normalise-parity.mjs shell out to rg, and scripts/gen-mcp-tools.mjs to fd. #1 resolves that by installing both binaries.

@tony tony changed the title Bound every CI job with a timeout Bound every CI job with a timeout, and install the audit tools Sep 7, 2026
@tony
tony force-pushed the github-actions-savings branch from be6395a to 33bd964 Compare September 7, 2026 02:24
tony added 2 commits September 7, 2026 19:57
GitHub's default job cap is six hours. The publish job in reusable-deploy.yml
ran 81.6 minutes against a 1.24-minute median before configure-aws-credentials
failed (run 34052068554); publish-root has a milder 11-minute instance.

Values are 2.5x p95 or 1.2x the observed maximum, whichever is larger, rounded
up to five minutes with a ten-minute floor.
Seven ports write a redirect to /reference/<slug>/ at <port>/<version>/api/.
py is excluded because that path is real gp-sphinx output, the oracle
check-style-parity.mjs measures against. But when upstream's checkout is
missing the Sphinx build is skipped and nothing is written there at all,
while the port index still links to it: 58 broken links to /en/py/latest/api/
on every runner, since ~/work/python/libtmux never exists in CI.

Fall back to the same redirect when the reference is not built. With the
checkout present the Sphinx output still wins, so the oracle is untouched.
@tony
tony force-pushed the github-actions-savings branch from 33bd964 to 8833d2b Compare September 8, 2026 00:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant