diff --git a/doc/audit-reports/2026-09-07.md b/doc/audit-reports/2026-09-07.md new file mode 100644 index 0000000000..ce08c1d416 --- /dev/null +++ b/doc/audit-reports/2026-09-07.md @@ -0,0 +1,140 @@ +# Constitution-vs-reality audit — 2026-09-07 + +Second run under the procedure in `doc/constitution-audit.md` (#535), and the first +fired by the now-active monthly cloud routine (`doc/scheduled-routines.md`, ACTIVE +since 2026-09-06). Audited `origin/main` at `a909676` ("settings: keep the Voices tab +in step with the current session (#227) (#636)"). Executed locally in a fresh +worktree (`.worktrees/constitution-audit-2026-09`); hermetic (no real-host runs, no +`.env.production`). + +**Verdict: zero genuine drift found. No issues filed.** Everything the constitution +claims checks out against reality, including every item flagged in the 2026-07-07 +report — the one drift item found then (#551, spec-inventory coverage) is not only +fixed but now enforced by an automated CI guard. 3 issues are correctly in-flight +behind `awaiting-founder` and are not drift. + +## Lens (a) — governance docs vs reality: PASS + +| Claim | Reality | Result | +| --- | --- | --- | +| `awaiting-founder` label exists, documented meaning | Exists, `#B60205`, "Blocked on a founder decision or sign-off — needs Ross's attention" | ✅ | +| `founder-approved` label (path-guard remedy) exists | Exists, `#0E8A16`, "Founder has approved this founder-gated change (path-guard gate)" | ✅ | +| `awaiting-founder-notify.yaml` @-mentions the founder on label apply | Confirmed by source: `pull_request_target`/`issues: labeled`, posts a comment `@rosscado — … awaiting a founder decision …` with the standing-queue link | ✅ | +| Merge gate = `npm test` CI (`test.yaml`) + required `e2e` check (`e2e.yaml`) | Both workflows match their documented behaviour exactly (source-read); last 8 pushes to `main` all show `e2e` conclusion `success` (run #580 down to #566, via the Actions API) | ✅ | +| Branch protection required contexts are exactly `["test (22.x)","e2e"]` | No direct branch-protection API available in this session's toolset; corroborated instead from two angles: (1) a recent PR's check-run list (`#648`) shows 5 checks running (`test (22.x)`, `e2e`, `e2e-firefox`, `gitleaks`, `path-guard`) with only the first two documented as required/gating; (2) open issue **#554**'s own body states the branch-protection API currently returns exactly `test (22.x)` + `e2e` — consistent, and #554's entire premise (promote path-guard/secret-scan) confirms they are *not yet* required | ✅ (indirect but consistent) | +| Founder-gated path enumeration (AGENTS.md) matches the mechanical guard | `scripts/path-guard-lib.mjs` now on `main` (PR #539 merged since the July report). Its `GATED_PATH_PATTERNS` — `src/JwtManager.ts`, `src/auth/`, `wxt.config.ts`, **`entrypoints/saypi.content.ts`**, **`entrypoints/saypi-universal.content.ts`**, release machinery, `.github/workflows/`, itself — now matches AGENTS.md's "High-blast-radius changes" wording verbatim, including the two content-script shims. The July report's forward-note ("align AGENTS.md's injection-scope wording with the shipped pattern list") was correctly applied when #539 landed. | ✅ | +| `path-guard` still advisory pending founder promotion | `path-guard.yaml`'s own header comment says so; issue **#554** (open, `awaiting-founder`) is the live promotion request, citing the same 5-consecutive-green bar used for the `e2e` precedent | ✅ (in-flight, correctly not required yet) | +| Spend ledger (#533): `.l4-ledger.json` gitignored, caps 6/12h + 25/7-day, `SAYPI_L4_CAP_SESSION`/`_WEEK`/`_OVERRIDE` | `.gitignore` covers the ledger; `DEFAULT_CAPS = { session: 6, week: 25 }` in `l4-ledger-lib.mjs`; `node scripts/l4-ledger.mjs report` runs clean (0/6, 0/25); all three env vars wired and used at all three call sites (`layer4cdp.mjs`, `e2e-host-sweep.mjs`, `e2e-dictation-sweep.mjs`) | ✅ | +| AGENTS.md's "PROPOSED defaults pending founder confirmation" framing for the cap *numbers* | Still accurate: issue #533 closed 2026-07-07 as *done* (ledger infrastructure shipped), but its closing comment explicitly left the 6/25 numbers as non-blocking and unconfirmed. No later comment confirms them. Not drift — the doc and the code agree the numbers are still proposed — but noting it here since the gap has now sat open (unlabelled) for two months without a formal ask. Not filed: no doc asserts confirmation happened. | ℹ️ note, not drift | +| `release:status` / `release:freshness` exist and support `--json` | Both in `package.json` → `scripts/store-status.mjs` (`--json`, `--freshness`); ran both locally — both degrade cleanly to `SKIPPED` per store (no credentials in this worktree), exit 0, matching the documented no-credential behaviour | ✅ | +| Credential rotation runbook (#534) | "Rotation & lifetimes (#534)" section present in `doc/release/publishing-credentials.md`, linked from `doc/release/README.md` | ✅ | +| Every doc `AGENTS.md`/`CLAUDE.md` reference exists | 20 checked (all Assistant/Agent Docs entries, `doc/release/delegation-criteria.md` and `doc/release/kill-switch-policy.md` included — both were in-flight in July, both now present) — all present | ✅ | +| Every npm alias a governing doc cites exists | Full `jq .scripts package.json` diffed against every alias named in `AGENTS.md`/`CLAUDE.md`/the harness docs — all present, none renamed | ✅ | +| Memory-vs-docs precedence line | Present (`AGENTS.md:154`) | ✅ | +| `.cursor/rules` staleness warning still accurate | Both `build_process.mdc` (webpack.config.js, `manifest.json` web_accessible_resources) and `vad_model_loading.mdc` (`webpack.config.js` glob, `manifest.json`) still describe the pre-WXT build — warning remains correct and necessary | ✅ | +| `doc/scheduled-routines.md` claims | Trigger ids, cron expressions, tool restriction (`Bash, Read, Glob, Grep, Edit, Write, TodoWrite`), "ACTIVE since 2026-09-06" — this file *is* the routine currently executing this audit, and its self-description matches observed behaviour (worktree isolation, PR-only, hermetic) | ✅ | +| `doc/codebase-caution-map.md` referenced and current | Exists, referenced correctly from both AGENTS.md and CLAUDE.md | ✅ | +| `awaiting-founder` queue | 3 open issues, 0 open PRs (see below) | ✅ listed | + +**Observation (not drift):** `.github/workflows/gitleaks.yaml`→`secret-scan.yaml` and +`e2e-firefox.yaml` now run on every PR (confirmed on #648's check-run list) but are +not named anywhere in `AGENTS.md`. This isn't a false claim — AGENTS.md never asserts +they don't exist — so it doesn't meet the drift bar, but it's a documentation gap +worth folding in whenever #554 (their required-check promotion) lands and touches +this section anyway. + +## Lens (b) — Layer-3 harness health: PASS + +- `npm ci` — clean install (1557 packages). +- `npm run e2e:build` — clean (dev-mode static build, manifest guard satisfied, + 57.16 MB output). +- `npm run test:e2e` (full Playwright suite) — **could not complete in this + session's sandbox**: `browserType.launchPersistentContext` failed with + `Executable doesn't exist at /opt/pw-browsers/chromium-1223/chrome-linux64/chrome`. + The pinned `@playwright/test@1.60.0` (exact, from `package-lock.json`) requires + Chromium revision 1223; this sandbox's pre-baked browser cache only has revision + 1194 (an older, differently-packaged build). `npx playwright install` is + off-limits per this environment's own instructions + (`PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1`). A diagnostic-only, uncommitted local edit + (swap `channel: "chromium"` for `executablePath: "/opt/pw-browsers/chromium"` in + `e2e/fixtures/extension.ts`, reverted immediately after) got the browser to launch + but then failed 45/48 specs on unrelated `chrome.storage`/extension-API errors — + consistent with the older Chromium build not supporting what the MV3 harness + needs, not with a real product or harness defect. **This is a sandbox + browser-provisioning gap in this particular audit session, not repo drift**, and + is not filed as an issue (there is nothing in the repo to fix — it is this + environment's pinned browser image lagging the pinned `@playwright/test` version). + No code changes were kept from the diagnostic (`git status` clean afterward). +- **Corroborating evidence the harness itself is healthy**: the `e2e` required check + is `success` on the last 8 consecutive pushes to `main` (run #580 back to #566, + spanning 2026-09-06's whole merge batch — 8 different PRs, no docs-only + exemptions), via the GitHub Actions API. CI's own `e2e.yaml` installs its browsers + fresh every run (`npx playwright install --with-deps chromium`), so it is not + exposed to this sandbox's stale cache. +- `npm test` (type-check + Jest + Vitest) — **green**: 262 files, **3024 passed / 1 + skipped**, ~101s. This exercises the same `.wxt/` regeneration path CI's `test` + job uses and confirms Layer 0–2 are fully healthy from a fresh checkout. +- **Spec inventory vs `e2e/README.md`** — now enforced by an automated guard, not + just a manual check: `test/e2e/readme-spec-inventory.spec.ts` (added by PR #648, + itself born from a same-day drift the guard caught between two independently-green + PRs — see #648's own PR description for the mechanism). Ran it directly + (`npx vitest run test/e2e/readme-spec-inventory.spec.ts`) — **5/5 passed**. Spec + count is now 22 (`e2e/specs/*.e2e.ts`), up from 12 in July; all 22 have README rows. +- README posture claims spot-checked: `settings.visual.ts` still documented and + wired as on-demand / NOT in the CI gate (own `playwright.visual.config.ts`, + `test:e2e:visual*` aliases, absent from both `test.yaml` and `e2e.yaml`) ✅; + fail-closed `MAP * ~NOTFOUND` host-resolver claim matches + `e2e/fixtures/launch-args.ts:28-34` exactly, including the "must be LAST" ordering + comment ✅. + +## Lens (c) — Layer-4 harnesses + ops tooling (static): PASS + +- All scripts the constitution points at exist: `layer4cdp.mjs`(+lib), `layer35.mjs`, + `dev-rig.mjs`, `e2e-host-sweep.mjs`(+lib), `e2e-dictation-sweep.mjs`(+lib), + `l4-ledger.mjs`(+lib), `store-status.mjs`, `release.mjs`. +- `layer4cdp` doc vs script: subcommands `seed | diagnose | verify | self-test` all + implemented (`layer4cdp.mjs:338-341`) with matching npm aliases; `--no-turn` and + `--headless` both present in the script and match the doc's flag examples. ✅ +- `layer35` doc vs script: subcommands `seed | verify | self-test` implemented and + aliased. ✅ +- Host-sweep script targets exactly pi.ai / claude.ai / chatgpt.com + (`e2e-host-sweep-lib.mjs:8-10`), matching the doc. ✅ +- Dictation-sweep `TARGETS` = `fixture` / `mistral` / `grok` + (`e2e-dictation-sweep-lib.mjs`), matching the doc's target table exactly. ✅ +- Ledger/caps wiring: verified once under lens (a); `SAYPI_L4_CAP_OVERRIDE` is + referenced (with a matching comment) in all three real-host entry points + (`layer4cdp.mjs`, `e2e-host-sweep.mjs`, `e2e-dictation-sweep.mjs`). ✅ +- Release/ops tooling: `npm run release:status -- --json` and + `npm run release:freshness -- --json` both ran locally, both degraded cleanly to + per-store `SKIPPED` (no publishing credentials in this worktree, as expected) with + exit 0; `doc/release/publishing-credentials.md` carries the "Rotation & lifetimes + (#534)" section `doc/release/README.md` points readers to; + `doc/post-release-monitoring.md`'s cited commands (`npm run release:status`, etc.) + resolve. ✅ + +## Drift found → issues filed + +None this cycle. + +## Fixed inline in the audit PR + +None needed — no drift found to fix. + +## In-flight (correctly awaiting founder — listed, NOT drift) + +`awaiting-founder` queue at audit time (issues only; 0 open PRs carry the label): + +- **#652** — Release candidate v1.15.0 ready for founder review (Stage 1 automated + prep; opened same-day by the 2026-09-07 weekly maintenance routine, 28 user-facing + merges since the v1.14.0 baseline). +- **#556** — Build the Stage-2 release workflow (one-click store release via + environment-scoped CI secrets); blocked on #554's promotions per its own + acceptance criteria. +- **#554** — Promote `path-guard` and `secret-scan` to required branch-protection + checks once each hits 5 consecutive first-attempt-green runs on distinct + non-docs-only PRs; gates Stage 2/3 of the release-delegation criteria. + +## Next audit + +First Monday of October 2026 (2026-10-05), 06:30 UTC via the now-active cloud +routine (`doc/scheduled-routines.md`).