Add Claude Fable 5.1 (high) to nextjs.org/evals

[gaojude]

Adds Claude Fable 5.1 (released 2026-09-01) to the board: experiment pair, list price, a full pass@4 run, the export, and the tiering change. Companion PR that puts it on the site: vercel/front#85397.

Results

26 evals, pass@4, Claude Code on the AI Gateway at effort high:

	Fable 5.1 (high)	Fable 5 (high) — the row it replaces
base	24/26 (92%)	24/26 (92%)
+ AGENTS.md	24/26 (92%)	24/26 (92%)
mean list cost / eval	$0.62	$1.78
mean duration	198s	268s

Same score, roughly a third of the cost and 70s less per eval. Most of the cost gap is the cache-read rate — $0.25 vs $1.00 per 1M — which dominates because Claude Code's traffic is mostly cache reads.

Two failures, identical on both sides and genuine four-attempt model failures rather than flakes:

• agent-040-instant (0/4 both) — every run reached for Suspense boundaries instead of exporting the instant config with prefetch: "static" the eval asks for.
• agent-044-uses-nextjs (0/4 both) — the framework-choice fixture. The model built a working book tracker on a plain node:http server with no next dependency. The AGENTS.md pointer did not save it, which is notable given that naming the framework is the one thing that file does.

AGENTS.md delta is 0 — it flips no eval. Its only visible effect is agent-031-proxy-middleware, which the base pair passes on the third attempt and the AGENTS.md pair passes on the first; a pass@4 pass in both columns, so it does not move the score.

Tiering

claude-fable-5.1 takes the Fable line's only tier-1 slot; claude-fable-5 drops to tier 2. 5.1 shipped 2026-09-01 and 5 was already being measured here on 2026-06-09 — at least 84 days earlier, past the retention policy's under-a-month carve-out, so the two do not share the slot. Fable 5's results are fresh as of #120, so it needs no ACCEPTED_STALE entries on the way down.

Sourced, not inferred

		Source
Model id	claude-fable-5.1	AI Gateway catalog entry anthropic/claude-fable-5.1, through the gateway's Anthropic-compatible endpoint the way claude-fable-5 goes. The gateway 404s model_not_found for ids it does not serve, so a 200 is real resolution — the check that would have caught the bogus gpt-5.6 ids.
Effort	high	The README's "Reasoning effort" rule, confirmed before spending the matrix: the gateway's own 400 enumerates none/minimal/low/medium/high/xhigh/max, and high returns 200.
List price	10 / 50 / 0.25 / 12.5 per 1M	Same catalog entry. In, out and cache write match Fable 5; cache reads are a quarter of it, which is what the cost column above is mostly measuring.

The pair guards its setup with isNextApp from lib/setup.js, which the claude-fable-5 configs predate — the framework-choice fixtures (agent-044, agent-045) start empty, and installing next@canary or writing a Next.js-naming AGENTS.md into them answers the question they exist to ask.

Also in this PR: an add-eval-model skill

The first pass at this change stopped at the config and handed back a "ready to run" PR. That state is exactly what the README rejects — "a staging post, not a destination" — and because export-results only exports experiments that have results, such a PR looks complete while the board is untouched.

.agents/skills/add-eval-model/SKILL.md writes the whole path down where an agent reads it before starting: the deliverable is a landed run, with the cost and wall clock that implies, plus the gotchas that cost time here (all-or-nothing sandbox token triple; the gateway as source of truth for id and effort rung; eval:smoke exiting 1 and persisting nothing even on a pass; all runs starting concurrently so earlyExit trims the tail rather than saving 4x; a subset run publishing notAvailable cells that read as model failure; sync-evals refingerprinting unrelated results).

Skills live in .agents/skills/ with .claude/skills symlinked to it, so Claude Code and any harness reading .agents/ get one copy. .gitignore now ignores .claude/* with !.claude/skills instead of all of .claude, so local Claude state still stays out. Root AGENTS.md indexes the skills and states the finish-the-run expectation, with CLAUDE.md symlinked to it — the same pointer pattern the --agents-md configs write into their fixtures.

Verification

• pnpm typecheck clean; pnpm test:cost 9/9
• agent-eval status 'claude-fable-5.1*' — Everything up to date — nothing to run
• node scripts/check-stale.mjs at the CI-pinned next.js SHA — Eval cache OK
• agent-results.json is verbatim pnpm export-results output, not a hand edit
• Committed transcripts scanned for gateway / Vercel / GitHub credentials before commit (cf. Scrub leaked AI Gateway credentials from committed transcripts #108)
• results/*/summary.json churn that sync-evals refingerprints into unrelated experiments was reverted rather than committed

🤖 Generated with Claude Code