Motivation
We're publishing these skills publicly (still experimental) and now support several agents (Claude Code, Codex, Gemini CLI, …). We need a repeatable way to tell whether a skill actually improves agent behaviour — both to judge a new skill and to catch regressions when we edit a SKILL.md (especially the description, which drives triggering). Today "does this work well?" is vibes-only.
Two things worth measuring
- Triggering / discoverability — does the
description make agents invoke the skill at the right time, and not fire on unrelated prompts? (false negatives + false positives)
- Task success — once triggered, does following the
SKILL.md produce a correct, well-cited result? e.g. the right chart is found, the right endpoint/params are used, per-capita joins are done correctly, citations are surfaced, and the analytics cost/anti-patterns are avoided.
Proposed approach (staged — start manual, then automate)
- Golden prompt set per skill: a handful of representative user prompts + expected outcome (which skill should fire, which chart/slug/endpoint, what the answer must contain). Store in-repo, e.g.
evals/<skill>/cases.yml.
- Grading: deterministic checks where possible (did it call the right endpoint? return the expected slug? include a citation?), plus LLM-as-judge against a short rubric for open-ended answers.
- Cross-agent runs: run the same golden set through Claude Code, Codex, and Gemini CLI — a description that reads well to one model can under- or over-trigger in another, and that's exactly the kind of thing we want to see.
- Cadence: run on PRs that touch a skill (regression gate) + a periodic full run. Keep it cheap — public endpoints, small judge model.
- Output: a per-skill scorecard (trigger precision/recall, task pass-rate) so "performing well or not" becomes a number we can watch over time.
Start small
Even a hand-run rubric with 3–5 prompts per skill beats nothing. Nail the shape first, automate second.
Open questions
- Bespoke harness vs. an existing skills-eval tool?
- How to drive non-Claude agents headlessly in CI?
- Gate PRs on the evals, or just report?
- Where does judge cost land?
Related
Filed with Claude Code at Lucas's request.
Motivation
We're publishing these skills publicly (still experimental) and now support several agents (Claude Code, Codex, Gemini CLI, …). We need a repeatable way to tell whether a skill actually improves agent behaviour — both to judge a new skill and to catch regressions when we edit a
SKILL.md(especially thedescription, which drives triggering). Today "does this work well?" is vibes-only.Two things worth measuring
descriptionmake agents invoke the skill at the right time, and not fire on unrelated prompts? (false negatives + false positives)SKILL.mdproduce a correct, well-cited result? e.g. the right chart is found, the right endpoint/params are used, per-capita joins are done correctly, citations are surfaced, and the analytics cost/anti-patterns are avoided.Proposed approach (staged — start manual, then automate)
evals/<skill>/cases.yml.Start small
Even a hand-run rubric with 3–5 prompts per skill beats nothing. Nail the shape first, automate second.
Open questions
Related
Filed with Claude Code at Lucas's request.