feat(verify): verify.syntax and verify.run — read-only checks against a throwaway copy (F5, F6, F35) - #432
Open
plombeer31 wants to merge 5 commits into
Open
feat(verify): verify.syntax and verify.run — read-only checks against a throwaway copy (F5, F6, F35)#432plombeer31 wants to merge 5 commits into
plombeer31 wants to merge 5 commits into
Conversation
…r a fake pass
New tool family `verify.*` under src/tools/verify/, starting with
`verify.syntax {files}`. One checker per file by extension, one process
per file (never `node --check a b c`):
- .js/.cjs/.mjs/.json: the write-time parse check in-process, then one
`node --check` for what a plain-script parse cannot judge (ES modules);
- .ts/.tsx: the project's own `tsc --noEmit -p tsconfig.json`, once per
project, diagnostics attributed per file; a file the project does not
include gets no verdict; no tsconfig or no node_modules/.bin/tsc → "no
checker";
- .py: python3 `compile()` (not py_compile — that writes __pycache__ next
to the deliverable); .sh/.bash: `bash -n`;
- .html: every inline <script> through the JS checker, plus a warning for
content after </html>; .css: brace balance outside strings/comments.
Per file `{file, ok: true|false|null, checker, error?, warning?}`; an
unknown extension is `ok: null, checker: "none"` and the summary says
`no checker for .x (N files)`. Unchecked never counts as passing.
Summary: failures first, then unchecked, then ok, ≤ 4,000 chars.
Registered like the os.* tools: descriptor (frequent tier), args JSON
schema, `verify-tool` in the GBNF tool-name rule, `pure_read` resource
class, bootstrap registration, worker read-scope confinement. Read-only,
so the fusion orchestrator gate lets it run (decision D1) — pinned with
the real tool definition.
…a throwaway copy
`verify.run {kind: "command" | "service" | "page", …}` executes what was
built and reports what happened, for any language or artefact:
- command: exit code, stdout/stderr tails (8,000 chars, tail first),
duration; a pre-joined command line goes through the OS subshell like
`os.shell.run`;
- service: spawn `start`, wait for `ready` (TCP port open or URL 2xx,
default 30 s), send `requests` with fetch ({status, ok, bodyHead, ms}),
kill the group;
- page: playwright-core with the configured browser channel /
executable, else the first Chromium-family browser found, else a
Playwright-managed one — always headless; when nothing launches the
result is `{ok: false, error: "no browser available: …"}`, never a
pass. Loads file://<copy>/<path> or `url`, collects pageerror,
console errors + warnings, request failures and null
getElementById/querySelector lookups (an injected wrapper), runs the
`script` steps, samples every probe `expr` every 250 ms for
`seconds` (≤ 40 samples per probe).
Isolation: every run happens in a copy under os.tmpdir()/atag-verify-<id>
— `cp -c -R` (clonefile) on macOS, elsewhere a copy with node_modules,
.git, dist, build, target, .venv symlinked in; a tree over 2 GB runs in
place with `isolated: false`. The copy is deleted afterwards. The
process is spawned detached and its whole group is SIGKILLed on timeout,
abort and completion. `network: false` (default) sets HTTP_PROXY /
HTTPS_PROXY / ALL_PROXY to an unreachable local port and empties
NO_PROXY — a soft block, documented as not a sandbox; the page runner
aborts http(s) requests outright.
`checks`: exit N | exit != N, stdout/stderr [not] contains "x",
status N, no errors, missing selectors N, probe <name>
decreases|increases|equals|reaches|stays. Unknown syntax fails with
`unknown check`; a run that did not happen fails every check as
`not evaluated`. Common result {ok, kind, isolated, durationMs, …,
checks} plus a summary ≤ 4,000 chars with failing checks and errors
first. Programmatic API: runVerify(args, ctx) and runChecks(specs, ctx)
for the fan-out contract.
Registered like os.*: descriptor with examples, args JSON schema (env is
map-shaped, so the strict converter refuses it — pinned), `verify-tool`
grammar rule, `approval_gated` resource class under the shell category
(asks below level 4 like os.shell.run, silent inside an authorised
fan-out scope), bootstrap wiring. `readonly: true`, so the fusion
orchestrator gate lets it run (D1).
`### fusion` gains one line — "Before accepting a fan-out, check it:
`verify.syntax` on the declared files and `verify.run` on what the
request must do." — with three sentences trimmed to stay under the
1,400-char budget; the pinned phrase test follows the new wording.
Pins the property the throwaway copy exists for, before and after, with a byte-level snapshot of the workspace: a `verify.run` command that writes a harness, appends to index.html, creates a directory and deletes a source file; a service that logs to its own directory; a run killed on timeout (the copy is still removed); and `verify.syntax` over .py/.ts/.html/.css/.js/.sh (no __pycache__, no tsbuildinfo, no temp file). The TypeScript checker had one hole: an `incremental` (or `composite`) project writes `tsconfig.tsbuildinfo` beside its tsconfig even under `--noEmit`. It now runs `--incremental --tsBuildInfoFile <os tmp>` and removes the file — legal for plain and composite projects alike. Out of scope, noted in the test header: `os.fs.trash` goes through Finder, which leaves a `.DS_Store` in the directory it trashed from.
…d steadier tests
`VerifyRunContext.config` is optional: `runChecks(specs, {workingDir,
signal})` — the shape the fan-out contract calls with — falls back to
`getConfig().browser` for page runs. `openai-strict-tools.test.ts` sweeps
every registered tool and lists the map-shaped refusals; `verify.run`
(its `env`) joins `os.http.request` and `mcp.prompt.get` there.
Two tests asserted cleanup by scanning the shared OS temp dir, which a
concurrent test process also fills; they now record the copy directory
through the `workspace` seam and assert on exactly that path. The
`node --check` fallback test gets a 60 s budget — two spawns took over
15 s under the full parallel suite.
Bring the tree to the state the package has once the other packages are merged around it: shared seams (config version, prompt tail order, fusion facts) resolved the same way as in the integrated build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Before:
node --checkwas refused for the orchestrator like a write, so a cloud orchestrator replied "Project Complete" over a game that threw on launch; workers hand-wrote verification harnesses into the deliverable; nothing in the loop ran what was built.After: a
verify.*tool family,readonlyso the fusion gate admits it.verify.syntax {files}runs one checker per file by type (JS/JSON in-process,tsc --noEmitwhen the project has it,python3 -m py_compile,bash -n, inline<script>blocks in HTML plus a warning on content after</html>, CSS brace balance) and answersno checker for .xinstead of passing.verify.runruns acommand, aservice(start, readiness, requests) or apage(headless browser, scripted input, probes; "no browser available" is a failure, never a pass) against a throwaway copy of the working directory (clonefile on macOS, exclude-and-symlink elsewhere;isolated: falsewhen the tree is too big) with a timeout, a killed process group and network off by default, and evaluates plainchecks(exit 0,stdout contains,status 200,no errors,probe x decreases, …).runChecks(specs, ctx)is exported and wired intofusion.delegateso a contract's checks run on a copy. One guidance sentence: check before you accept.Why
Design call D1: the orchestrator may verify, read-only; it still never builds. Four non-working benchmark turns broke only at runtime and passed every static check; each would have been caught by one
commandorpagerun. A checker that silently passes an unknown file type would be worse than none. Includes the reconcile commit that wiresrunChecksinto the delegate seam and exemptsverify.runfrom the orchestrator-role purity pin (it is approval-gated below level 4 yet read-only for the gate because it runs on a copy).How it was verified
npm run lintcleannpx vitest run --minWorkers=1 --maxWorkers=3 src/agent src/llm/grammar src/llm/provider/openai src/prompt src/runtime/bootstrap.test.ts src/tools/fusion src/tools src/tools/verify— 191 files / 2447 tests green, 3 skipped (baseline failures: none)verify.syntaxunprompted before its first replyStacked on #431; merge in order.