Publish BBEH calibration results and evidence assessment - #77
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a232d39ffa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if hashlib.sha256(row['response'].encode()).hexdigest()!=row['response_sha256']:raise ValueError('BBEH response hash mismatch') | ||
| if namespace['evaluate_correctness'](row['response'],row['reference'])!=row['correct']:raise ValueError('BBEH score mismatch') |
There was a problem hiding this comment.
When a generated report contains stale or miscalculated calibration.scores, this function still accepts it because it replays only each outcome's correct field and never recomputes the aggregate scores, counts, or bounds. publish.py then refreshes the digest, and bbeh-page.py renders the unchecked aggregate, so a publication can pass while displaying a score inconsistent with its underlying outcomes; derive or validate every published aggregate from the replayed rows.
AGENTS.md reference: benchmarks/site/AGENTS.md:L11-L14
Useful? React with 👍 / 👎.
| m=verify(root);assert not (root/'calibration-freeze.json').exists(),'Already settled; do not restart unchanged' | ||
| assert (root/'offline-check.json').is_file() |
There was a problem hiding this comment.
For a fresh init/run, execution always stops here unless the user manually invents offline-check.json: a repository-wide search for offline-check finds no command or test that creates it. Moreover, only existence is checked, so an empty file bypasses the intended scorer, isolation, and accounting preflight; generate a structured receipt from those checks and verify its contents or digest before dispatching paid calls.
Useful? React with 👍 / 👎.
1 participant
Runs the approved bounded BBEH Mini calibration and publishes its failed suitability gate with a source-bound assessment. Terra scored 3/12; Sonnet returned five originals with one correct and seven timeouts. The main Co-Evolution comparison was not launched, saving 168 conditional calls.
The report keeps missing outcomes unavailable and explains eight angle-bracket-only official failures without changing scores. It includes the pinned official scorer, immutable input/response provenance, incomplete cost accounting, and a separate page in the existing observatory style. Publication replays the scorer and rejects altered response bytes or stale assessments; existing studies and archives remain intact.
Validation: seven publication tests, two calibration/key-isolation tests, official scorer replay, archive integrity, and desktop/mobile visual checks passed. No extra model calls were made for publication.