Conversation
why: Nothing ran the examples the site quotes. A port could break the code a page shows while every docs check still passed, and each port's arena adapter had no supervisor outside its own tests to lend it a server. what: - Add scripts/arena/supervisor.mjs: start tmux with -D -S and an empty config in a private root, authenticate its PID, install a challenge, lend the server under the arena contract, require one evidence record naming the live challenge, PID, and socket, recheck the server, and clean up - Add scripts/arena/artifacts.mjs with each port's build and run recipe from its tmux-arena worktree and the sources each artifact executes - Add scripts/docs-arena.mjs (pnpm test:arena): report quote coverage, run each available port, then withhold the socket and require the adapter to fail closed - Add scripts/docs-arena.negative.mjs and a fixture adapter proving the supervisor rejects each way an adapter can fake contact - Run the negative in the publication audit and install tmux in CI; run the port lane only with LIBTMUX_DOCS_ARENA=1 - Document the command in CONTRIBUTING.md
why: docs-arena counted a quoted source as run by matching a `slug:path` key, and a key says nothing about content. A port's `tmux-arena` worktree is free to diverge from the `-docs` worktree the site quotes, and nothing noticed — two of the four quoted-and-run sources already had. Coverage also counted doctest pages and quoted files in one bucket, where a page can never have a quoted-file entry at all, so a page read as a file nothing quotes and a file the arena runs but no page shows read as covered. what: - Add `scripts/arena/check-quote-drift.mjs`: for every source an artifact both quotes and runs, compare the two, region-aware. A file may differ outside the region a page quotes — evidence plumbing wrapped around a reader-facing body is exactly that — but the region must match, and a region missing from the run side is its own failure. - Export `sliceRegion` from `remark-port-code.mjs`, so the check slices a region with the same matcher the page does. One implementation, so a region cannot mean two things depending on the caller. - Record the two mismatches it finds in `KNOWN_DRIFT` with the reason. A listed key reports rather than fails; an unlisted mismatch fails, and a listed key that now matches fails until its entry goes, so the list shrinks. - Report coverage in three buckets: quoted and run, quoted with no adapter, and run but quoted by no page. Pages are counted separately. - Wire it into `test-all.sh`: the negative everywhere, since it reads files and nothing else; the live comparison in the port lane, which is where the worktrees are. An absent worktree reports as not run, never as drift.
…here why: the first run of this check against a newly registered artifact reported swift's `Waiting.swift` as drift, and it was not. A region marker lives on the port's `docs-site` branch and never on the branch the arena runs from — that is the whole reason markers are kept off the release branch — so the run side has no marker to slice by, and requiring one made the rule unsatisfiable for every region-scoped quote. what: when the quoted side has the region and the run side has no marker, compare the lines the page shows against the file instead, in order. A page showing code that is not in the file still fails, and the message says the comparison was made with or without markers. Two negative cases: the same lines without markers passes, changed lines fails.
why: four of the Go port documented Examples failed against a lent server and passed everywhere else, each after exactly sixty seconds with empty output. The cause was not the examples. The supervisor gives the server a private HOME, and a HOME with no startup file is a new account to an interactive shell: zsh answers with its first-run configuration wizard, so the pane waits for a keypress and prints nothing, and an example that sends keys and reads the pane back times out having captured nothing. what: write empty `.zshrc`, `.zshenv`, `.bashrc` and `.profile` into that HOME. python's pytest fixture already writes one for this exact reason, which is where the diagnosis came from.
why: an artifact that runs several documented sources — a doctest page is that shape — had to be lent one server per source, because the contract allowed exactly one evidence record per run. Five sources cost about 421 ms that way against about 85 ms in a single lend. python's gate already stamped a `source` field naming the page it ran, and nothing read it. what: - Add `parseEvidenceRecords`, `validateEvidenceBySource` and `runInArenaMulti` beside the single-record `parseEvidence`, `validateEvidence` and `runInArena`, which are unchanged and still what every artifact uses today. A run is rejected when a record names an undeclared source, names one twice, carries no source, or when a declared source produced none. - Prove a server replaced partway through is caught: the fixture stops the lent server between sources and lets another start on the same socket, and the run fails. That is what a documented block did to python's gate, where the replacement quietly answered the rest of the page with no challenge set. - Name the private socket root with `LIBTMUX_DOCS_ARENA_ROOT`, so a run given a sandbox cannot start a server outside it; `TMPDIR` still applies otherwise. - Run the new negative suite in `test-all.sh`, beside the arena negative.
why: six of the eight artifacts are named `<language>-<example>`. The other two carried a name from their build system instead — a CMake target and a .NET example case — so the registry read as if two ports had a different convention rather than two names that had never been chosen. what: `csharp-one-shot` and `cpp-tour`, matching the ids those ports now report. Both were run against this registry before the change landed.
why: the registry had one entry per port, so a port with four quoted examples proved one of them. The other three were shown to readers without ever having been run against a lent server. what: - Three more ts entries: capture, agent and workspace, each naming its own artifact, test file and quoted source. They keep the `ts` slug, so `--port ts` and the worktree override still select all four. - Run each distinct prepare command once per worktree. Four entries declaring the same install and build otherwise paid for it four times. - The workspace example imports the published `@libtmux/workspace`, so its entry builds that package too. - Report how long each artifact took, which is what showed the repeated builds.
why: the C++ port can now lend its server to more than the tour, and an artifact the registry does not name cannot be proved. what: entries for cpp-workspace, cpp-readme and cpp-streaming, each building its own target. The configure step repeats in every entry and runs once.
why: three ports can now lend their server to every example they document, and an artifact this registry does not name cannot be proved. The site quotes four swift examples and nine go ones; it could prove one of each. what: entries for swift-changing, swift-waiting, swift-workspaces and swift-mcp-embedding, each filtered to the one test that runs its example against the lent server, and for the eight remaining go examples. go-workspace builds from the workspace module rather than the examples module, because that is where the source the site quotes lives.
why: the Java port can now lend its server to every example it documents, and an artifact this registry does not name cannot be proved. what: entries for java-find-panes-running, java-serve-tmux-over-mcp, java-watch-pane-output and java-watch-what-changes. The Gradle classpath step is identical in each and runs once per worktree.
why: the Rust port can lend its server to a second example now, and an artifact this registry does not name cannot be proved. Its other examples stop or assume sole possession of their server, so they stay owned-only. what: an entry for rust-find, built the same way as rust-inspect.
why: the .NET examples can each take a lent server now, and the entry here still ran the old one-example flag, which no longer exists. what: the flag becomes --arena <id>, and four more entries cover the other documented snippet files. Mcp.ConnectToSelectedSurface is left out on purpose: it starts a separate MCP process that resolves its own socket, so it would report evidence for a server it never used.
why: an artifact could only ever be one source. A documentation page or a file of documented Examples is many, and proving them one server at a time costs a server per source — about 421 ms for five, against about 85 ms for one lend. what: - An entry may declare `sources`. When it does, the run is lent one server and must answer with one evidence record per declared source, so a source that produced none is a failure rather than a quiet pass. One source stays one record, which is what every other artifact does. - `python-workspace-and-location` audits two documentation pages on one lend. - `go-tmux-examples` runs the library's own documented Example functions, 38 of them, on one lend.
why: the drift check asks, of the sources an artifact runs, whether the page shows the same bytes. Nothing asked the other direction, and that is the one a new page gets wrong: fencing a file no artifact has ever executed renders exactly as well as one tested every run. It was printed as "quoted with no arena adapter yet" and the run still passed, so the list of them could grow without anyone deciding to let it. what: - `check-quote-coverage.mjs` requires each quoted source either to be run by an artifact or to be listed in `NOT_IN_THE_ARENA` with a reason code from the shared table and the gate that does run it. An unlisted one fails; so does a listed one an artifact now runs, or that no page quotes any more, so the list can only shrink. A code outside the table fails too, since a one-off string there is how a table stops meaning anything. - It reads the two data files and nothing else, so it runs in the gate every job executes rather than the port lane. A page is written in a checkout with no ports, which is where it has to fail. - `docs-arena.mjs` reports and enforces the same rule from the same module, so the two lanes cannot disagree about it. The two entries are rs examples, both executed by the rs example runner. The throwaway-session example builds a server on a socket path it chooses, asserts no session survives, shuts it down and unlinks the socket, so lending it a server would stop the server and the assertion would be about the supervisor's sessions. The read-only MCP example serves the protocol on stdout, which is where the arena reads its evidence line. Verified: the check reports 14 quoted sources run, 2 exempt, none unaccounted for. Its negative covers a quoted source nothing runs and nothing excuses, an exemption an artifact now runs, an exemption no page quotes, an invented reason code, the `platform:` prefix in both directions, and three controls including the registry this repository ships.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
libtmux.org quotes code out of eight port repositories. Nothing on this side ran any of it, so a page could show a program that stopped compiling two releases ago and the build would be just as green.
This adds the docs-side half of the arena: a supervisor that owns a tmux server, lends it to each port's adapter, and requires proof the adapter reached that server.
What it does
scripts/docs-arena.mjsruns each registered example through its port'stmux-arenaworktree against a server this repository starts.scripts/arena/supervisor.mjsowns the server, endpoint, deadline and cleanup; an adapter that reports no evidence, the wrong artifact, a guessed challenge, another socket, a stopped server, a nonzero exit, a missed deadline or a stray second server is rejected.scripts/arena/artifacts.mjsregisters 37 artifacts across all eight ports.Two checks answer questions a key match cannot:
slug:pathkey says nothing about content, and two ports' arena worktrees had already diverged from what the site shows. The comparison is region-aware, because a file may legitimately differ outside the region a page quotes.Verification
Every check ships a negative suite, because a gate that cannot fail is worse than none:
docs-arena.negative.mjs— 13 cases, one per way an adapter can fake contact, plus a conforming control.docs-arena.ev.negative.mjs— the multi-source path, including a lent server stopped between sources and silently replaced.check-quote-drift.negative.mjs— both directions of the known-drift list.check-quote-coverage.negative.mjs— a quoted source nothing runs, an exemption an artifact now runs, an exemption no page quotes, an invented reason code, and theplatform:prefix.Live:
quote coverage: 14 quoted source(s) run by the arena, 2 exempt, 0 unaccounted for. Every artifact was run through the real supervisor and each failed closed when its socket was withheld.Two findings from building it
A 60-second timeout was zsh's new-user wizard. The supervisor gives each run a private
HOME, and an empty one makes zsh run its first-run configuration prompt, which waits on a terminal that is not there. Writing empty startup files into thatHOMEtook go's entry from over 180s to 3.6s.A rule that could not be satisfied. The drift check required a region marker on both sides, but a marker lives on a port's
docs-sitebranch and never on the branch the arena runs from — so every region-scoped quote would have failed. It now compares the lines a page shows against the file when the run side has no marker; a page showing code absent from the file still fails.The coverage check needs only two data files, no worktree and no toolchain, so it runs in the lane every CI job executes rather than the port lane — which is the point, since a page is written in a checkout that has no ports.