Add a behaviour eval layer built on claude plugin eval

[Marigold]

claude plugin eval runs a plugin against realistic prompts and then runs the same prompts again with no plugin loaded, reporting the difference. That second arm answers the question evals/README.md's layer 3 was written to ask and had no automated answer for: did the skill do this, or would Claude have managed anyway. This adds one worked case, a make behaviour target, and the guidance for writing the next one. No skill content changes.

Skim the code — it's one case, one make target and a manifest. The prose in evals/README.md is the actual deliverable and is where review effort is worth spending.

How it works here

A case is a directory under evals/; these live at evals/skills/<skill>/<case>/ so they sit with that skill's other eval inputs. prompt.md holds the prompt and the run's limits, each file under graders/ is one pass/fail check. The runner already ignores lib/, results/ and the per-skill directories, since none of them contain a prompt.md.

make behaviour grants exactly WebFetch(domain:ourworldindata.org). Narrow in both directions on purpose: the cases need it, and the baseline arm gets it too, so a positive Δ is the skill's doing rather than the grant's. Granting Bash would be closer to how these skills really run — they document curl and jq — but it puts every command under Claude Code's OS sandbox, whose preconditions are machine-dependent; see the appendix.

.claude-plugin/plugin.json is new and load-bearing. Without a plugin manifest the command resolves no plugin, silently evaluates plain Claude Code, and reports a Δ that means nothing. It carries no version, like everything else here, so claude plugin validate now emits one warning — documented in AGENTS.md so nobody "fixes" it by adding a version.

The case took three attempts, and that's the point

Two of the three are written into evals/README.md because they are exactly the failure mode this tool exists to expose:

Attempt	Why the no-plugin arm also passed	Δ
"Link me a life-expectancy chart"	The slug is famous; any model recalls it	0.00
An obscure chart, asked for as a map	With WebFetch granted, baseline Claude just browsed the site	0.00
Same prompt, plus a grader on the route	Only the skill knows the documented /api/search endpoint	+0.33

The shipped case keeps all three graders. The two answer graders pass in both arms — they are regression guards on the ?tab= mapping the skill documents, not evidence the skill helps, and the Δ says so honestly.

Verified

Ran the case at the default three runs per arm:

CASE              WITH  W/OUT Δ      RUNS COST
finds-a-map-link  1.00  0.67  +0.33  6    $0.78

make validate passes (with the documented version warning). make test and make triggers are untouched by this change and were not re-run.

Not verified: CI. This adds no workflow — running it on every PR would spend real money per run, and which cadence is worth it is a decision, not a default.

Decisions worth a second look

Layers 2 and 3 now overlap. A tool_used: Skill grader asks the same question as a trigger eval, on one prompt instead of ten, and claude plugin eval is first-party where run-trigger-eval.py is ~400 lines of ours. I did not fold layer 2 in — ten prompts per skill measures routing at a width one case can't, and the call is easier to make with a few more behaviour cases in hand. It's noted as an open question in the caveats rather than silently left.

evals.json is now legacy. Each skill still has one, holding hand-run cases in the agentskills.io format. I left them and added a line saying to port a case into the directory format when you next touch that skill, rather than doing four ports in a PR about the mechanism.

Context for whoever picks this up

• Granting Bash fails on a Mac with Docker Desktop. The sandbox refuses with "the Docker (~/.docker, DOCKER_CONFIG) credential store on this machine holds a symbolic link inside it", and the case records a run error rather than a score. Pointing DOCKER_CONFIG at a clean directory does not help — it checks ~/.docker regardless. This is why the case uses WebFetch.
• A run cannot read the eval directory, so a case can't leak its graders to the agent under test.
• file_exists only sees files Claude created, not ones it edited.
• Results land in evals/results/<timestamp>/ with a self-contained report.html showing each grader's verdict per run. Already gitignored.
• Requires Claude Code ≥ 2.1.269; measured on 2.1.278.

🤖 Generated with Claude Code