Publish PlanBench outcome and require test assessments - #73
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e811f238e2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| content='<p class="eyebrow">PLANBENCH BLOCKSWORLD HARD · 50-TASK SUBSET</p><h1>Readiness failed.<br>No benchmark score.</h1><p>The official validator worked, but one Fable critique in the excluded smoke test received a provider safeguard refusal. The frozen readiness gate stopped the run before the 50 scored tasks began.</p><div class="notice"><strong>0 of 200 benchmark plans evaluated.</strong> This is missing measurement, not a score of zero and not evidence that co-evolution helped or hurt.</div><h2>The planned comparison</h2><div class="table-scroll" tabindex="0" role="region" aria-label="PlanBench arm coverage"><table><thead><tr><th>Arm</th><th>Workflow</th><th>Evaluated</th><th>Score /100</th></tr></thead><tbody>' | ||
| for arm,s in result['scores'].items():content+=f'<tr><th scope="row">{arm}</th><td>{esc(s["workflow"])}</td><td>{s["evaluated"]}/50</td><td>Unavailable</td></tr>' |
There was a problem hiding this comment.
When planbench-results.json is refreshed with a partial or completed run, this renderer still publishes “Readiness failed,” “0 of 200,” and Unavailable for every score; the spend split, timing, and failure details below are likewise hard-coded. Updating the result and its assessment digest therefore passes validate-publication.py but regenerates a page that contradicts the new evidence. Render these values conditionally from completion, scores, spend, timing, and failures, or reject unsupported result states.
AGENTS.md reference: benchmarks/site/AGENTS.md:L11-L14
Useful? React with 👍 / 👎.
1 participant
The authorized PlanBench attempt stopped at readiness after Fable refused one smoke critique. This change publishes the outcome honestly: 11 calls spent, seven excluded smoke plans validated, and zero scored benchmark tasks. It includes the bounded runner, frozen result export, and an assessment that explains why no efficacy conclusion is available.
Every current result publication now requires a source-bound assessment of the question, findings, coverage, test quality, limitations, practical conclusion and next action. The GitHub Pages deployment rejects stale or missing assessments and verifies they are actually published. Existing coding and custom-planning snapshots have assessments; archived editions remain unchanged.
Validation: official validator valid/invalid/malformed fixtures; one offline retry/resume lifecycle test; two excluded live smoke tasks; five publication-contract tests; six existing observatory tests; desktop/mobile rendering. No scored model calls followed the refusal, no model fallback was used, and the 336-call grant was not enlarged.