Run three Sonnet Terra benchmark screens and publish assessments - #80
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c43b6ed36
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if m['kind']=='gym':limitation+=' Generated tasks are a custom public-framework subset. Countdown success additionally requires an exact rational expression using only allowed operators and each number once; this stricter check is disclosed separately from upstream reward.' | ||
| if m['kind']=='bcb':limitation+=' Selection is conditional on the reference solution passing in the pinned local container; excluded tasks are disclosed in the eligibility manifest.' | ||
| assessment=dict(question='Does Terra critique followed by Sonnet revision improve on plain revision, matched Sonnet self-review and Terra alone?',finding=finding,test_quality='Pinned benchmark sources, container images, prompts, settings and task splits were recorded before calls. Offline valid/invalid fixtures and controller failure tests passed. All stage responses froze before grading; no hidden tests or answer keys reached participants. Response hashes and task-level grader receipts are preserved.',limitation=limitation,decision=decision,next_action='Publish this outcome and its limitations. Do not rerun failed or wrong answers or change settings after seeing scores. Consider a separate replication only if the predeclared practical gain and overhead thresholds are met.') | ||
| result=dict(schema='compact-results/1.0',benchmark=m['kind'],title=NAMES[m['kind']],completion=terminal['completion'],models=m['models'],effort=m['effort'],timeout_seconds=m['timeout_seconds'],source=m['source'],phases=phases,gate=gate,contrasts=contrasts,resources=resources,assessment=assessment,spend=dict(calls=len(charges),cap=204,by_provider=dict(Counter(r['seat'] for r in charges)),known_list_equivalent_usd=sum(r['cost_usd'] for r in charges if r['cost_usd'] is not None),unpriced_calls=[r['id'] for r in charges if r['cost_usd'] is None],attempts=charges),provenance=dict(manifest_sha256=digest(root/'manifest.json'),runtime_hashes=m['source_hashes'],receipt=load(root/'controller.exit.json')),finished=terminal['finished']) |
There was a problem hiding this comment.
The reports retain only runtime hashes, but the corresponding source is absent: all three exports identify campaign.py, support.py, and transport.py hashes that do not match any committed version, and the LiveCodeBench export also identifies a different run.py than the committed file. Because the referenced local run directory is ignored, reviewers cannot inspect or reproduce the controller, transport, retry, and gate logic that produced these published assessments. Include the frozen runtime files in the evidence or commit the exact historical versions alongside the reports.
AGENTS.md reference: AGENTS.md:L148-L150
Useful? React with 👍 / 👎.
1 participant
Runs the three approved Sonnet/Terra suitability screens and publishes every outcome with an assessment. All main phases stopped at their predefined gates: LiveCodeBench and Reasoning Gym had incomplete Sonnet baselines after two timeouts each, while BigCodeBench's Sonnet baselines exceeded the 75% ceiling. Co-Evolution efficacy remains unmeasured.
The harness freezes source versions, task cohorts and responses; uses isolated subscription transports and transactional reservations; stops new dispatches after irreversible calibration failure; and scores in pinned, network-isolated containers. The detached supervisors all exited normally. BigCodeBench selection followed reference checks (142 eligible of 148); Countdown scoring adds a disclosed exact expression check to enforce the task contract.
74 calls were used, with no main comparison calls. The $1.704913 known list-equivalent estimate is incomplete because four timed-out calls are unpriced. Existing studies and archives remain intact. LiveCodeBench-Pro was not triggered because LCB stopped on timeouts/runtime rather than saturation.
Validation: three controller tests, invalid/valid checker fixtures, reference checks, nine publication tests, archive hashes, and desktop/mobile visual checks. CI includes the controller and publication contracts. Public reports retain task-level answers, grader receipts, usage and provenance; unused main answer keys stay local.