Skip to content

Add Pixel Canary eval results - #128

Merged
quuu merged 1 commit into
mainfrom
codex/pixel-canary-evals
Sep 25, 2026
Merged

quuu merged 1 commit into
mainfrom
codex/pixel-canary-evals

Conversation

@ZYPX

@ZYPX ZYPX commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Adds Pixel Canary (stealth/pixel-canary) through Vercel AI Gateway and OpenCode, with baseline and AGENTS.md experiments, saved results for all 31 current evals, and the regenerated leaderboard export. Registers it as tier 1 without displacing an existing model; other models' exported data is unchanged.

Setup Passed Success rate Mean cost per eval
Baseline 28/31 90.3% N/A
With AGENTS.md 30/31 96.8% N/A

Both configurations use four attempts with early exit and a 2,400-second attempt limit. They reuse the existing pinned OpenCode binary with model metadata supplied through extraProviders. Reasoning effort uses the Gateway default; pricing remains unavailable.

Scoring exception: baseline agent-048-prefetch-url-data retains three genuine validation failures and one 900-second AI-judge timeout as an acknowledged failure. The 28/31 score therefore includes an ungraded timeout, rather than four genuine model failures for that eval. The classification and retry provenance record this explicitly; the cancelled regrading attempt is excluded.

The AGENTS.md variant passes refresh-settings and prefetch-url-data, with no regressions. Its only failed eval is uses-nextjs.

Validation:

  • pnpm typecheck
  • pnpm test:cost — 9 tests passed
  • node scripts/check-stale.mjs — Eval cache OK
  • pnpm export-results --check
  • Verified full coverage, unchanged exports for other models, and no configured credentials in the new result files.

@ZYPX
ZYPX requested a review from quuu September 25, 2026 22:12
@quuu
quuu merged commit 38cb8ad into main Sep 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants