Publish Sonnet/Terra PlanBench results and assessment - #76
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c5a88d8fcc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "data_sha256": "d979fd0071f828c63a0999f7165e168bacc53f81694e35384d73131bc7d2e250", | ||
| "question": "Does Terra review improve Sonnet plans beyond plain revision or independent Sonnet self-review on the same fixed PlanBench subset?", | ||
| "finding": "Sonnet originals scored 48/50 (96%). Plain revision, Sonnet self-review, and Terra review followed by Sonnet revision each scored 50/50 (100%). The two original plans executed legally but failed to satisfy the complete goal; all three revision approaches fixed them. Terra review added 0 points over either revision control and 4 points over the original draft.", | ||
| "test_quality": "All 200 scored plans were produced and evaluated after generation froze. The same pinned 50 tasks, original drafts, role prompts, upstream PDDL extractor and official VAL were used across arms. The two failed originals were goal failures, not parsing errors. Model routing, budget allocation, retry accounting, and exact exported plan hashes were checked; no validator feedback reached participants.", |
There was a problem hiding this comment.
The controller writes generation-freeze.json only after both dispatch loops finish (run.py lines 205–207), so the plans were produced before the freeze and only evaluated afterward. This assessment currently claims that production also occurred after the freeze, reversing a key evidence-control sequence; reword it to state that generation completed and froze before scoring.
AGENTS.md reference: benchmarks/site/AGENTS.md:L3-L8
Useful? React with 👍 / 👎.
1 participant
Publish a separate Sonnet/Terra run on the same fixed50 PlanBench Hard tasks. Sonnet originals scored96%; plain revision, self-review and Terra review each scored100%. Plain revision fixed both original goal failures without a reviewer. The assessment reports no added reviewer accuracy, near-ceiling limitations and the cost tradeoff: Terra review costs19% more than plain revision but20% less than Sonnet self-review.
The controller now routes author/reviewer roles and caps correctly across providers. A readiness-only amendment moved both models to medium effort and raised Claude's combined response allowance before scored testing, preserving six earlier calls. The final medium stage completed all200 plans;318/336 calls overall. Earlier Astra/Fable results and archives remain intact. The public report includes exact action sequences, paired evidence, resources, profile differences and a source-bound human-readable assessment.
Validation: role/cap/routing and retry tests, frozen official scoring of200 plans, publication/observatory checks,208 exact exported plan hashes, and desktop/mobile inspection. No scored output was used to tune prompts or choose the medium profile. Known list-equivalent cost is$7.29 with one early timeout unpriced.