feat(adapters): Superpowers skill evaluation adapter#134
Conversation
Add adapter to evaluate Superpowers skills against synthetic scenarios: - `skillopt_sleep/adapters/superpowers.py`: SuperpowersEvaluator class - Embedded scenarios for verification-before-completion skill - Rule-based judge (contains, regex, order, any_of ops) - Isolated HOME per scenario for clean state - Returns score compatible with SkillOpt gate - `tests/test_superpowers_scenarios.py`: Offline tests for judge logic Usage: from skillopt_sleep.adapters.superpowers import SuperpowersEvaluator evaluator = SuperpowersEvaluator(skill="verification-before-completion") results = evaluator.evaluate(candidate_skill_path) Refs microsoft#132 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add 2 more scenarios: partial-pass-honest, flaky-verify-rerun - Add token_cap parameter to SuperpowersEvaluator - Add total_tokens and total_latency_ms to EvalResults - Estimate tokens from output length (~4 chars/token) Now 5 scenarios testing verification-before-completion skill. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Copy candidate skill into temp superpowers copy, not original checkout - Add assertion: resolved skill path must be under workspace - Stamp each scenario result with pinned_sha, candidate_hash, scenario_seed - Clone superpowers at pinned SHA per evaluation run - Add --sha CLI flag Addresses feedback from truongsontung on microsoft#132.
- Use is_relative_to() instead of string prefix for path check - Raise ValueError instead of assert (survives -O) - Use git init+fetch+checkout instead of clone-then-fetch (no wasted download) - Add check=True to all git subprocess calls
Yif-Yang
left a comment
Yif-Yang
left a comment
There was a problem hiding this comment.
Thank you for picking up #132 and for iterating on the overlay/provenance ideas. There are tests here: we ran the 14 offline tests and the full suite (274 passed / 6 skipped). The current tests validate the rule matcher and scenario metadata, but they do not execute or validate the Superpowers integration path yet.
A clarification from our side: --target-skill-path is a SkillOpt-Sleep CLI option, not a Claude Code CLI option. The issue discussion did not make that distinction clearly enough. In the current adapter, Claude exits because it does not recognize that flag, so the candidate is not evaluated. The candidate is also copied to skills/SKILL.md rather than skills/verification-before-completion/SKILL.md, and the pinned Superpowers checkout is cloned but never loaded through its normal bootstrap/plugin/harness integration.
Before this can demonstrate Superpowers support, could you please add:
- A real, supported Superpowers harness path using the pinned checkout and normal bootstrap/plugin integration, with only
skills/verification-before-completion/SKILL.mdoverlaid in the temporary copy. - Fail-closed handling for non-zero Claude/git exits, timeout, malformed output, and missing scores. Please avoid inheriting unrelated host credentials and avoid unconditional
--dangerously-skip-permissions. - Deterministic mocked integration tests that exercise the actual runner/overlay path, prove the candidate file is the one loaded, and prove the source checkout remains unchanged.
- One opt-in real Claude Code smoke comparison (not public CI): baseline versus candidate with identical model/settings/tasks, raw outputs, and evidence that normal Superpowers behavior was active. A result showing no improvement is completely acceptable; the important part is proving the integration is real and reproducible.
- Corrections to the synthetic scenarios: the math tests currently reference
addwithout importing it, and the flaky helper setsTEST_RUN=1before its first run, so those scenarios do not test the intended behaviors. - Please remove the unrelated
CONTRIBUTING.mdpolicy change and the new 5,906-lineuv.lockunless a dependency change actually requires it, and update the PR description to match the current five scenarios/token fields.
As written in #132, testing a standalone skill-like prompt is not equivalent to running Superpowers. Once the real smoke and integration tests pass, we will be very happy to re-review and merge this through your original PR so your contribution is fully acknowledged.
- Use real harness path: skills/<name>/SKILL.md + HOME/.claude/skills symlink - Remove nonexistent --target-skill-path flag - Fail-closed on non-zero exit, timeout, missing claude binary - Fix scenarios: add missing imports, fix flaky test sentinel logic - Add 6 mocked integration tests proving overlay mechanism works - Add smoke_superpowers.sh for manual baseline/candidate comparison - Remove unrelated CONTRIBUTING.md change and 5.9k-line uv.lock Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
Thanks for the substantial update. The candidate overlay path, invalid CLI flag, fail-closed process handling, scenario fixtures, mocked integration coverage, and unrelated-file cleanup are all meaningful improvements. We re-ran the branch: the 20 focused tests pass, and the suite also passes when combined with current A few issues still block merging the adapter as a reliable Superpowers evaluator:
The feature direction remains valuable, and we would like to merge it through your PR so the contribution stays fully attributed. Once these focused correctness, live-integration, and safety points are addressed, we will re-review promptly. |
- Fix not_contains to split on pipe (all alternatives must be absent) - Add regression tests for false completion claim detection - Scrub host env: only PATH/TERM/LANG/ANTHROPIC_API_KEY, no credentials - Remove unconditional --dangerously-skip-permissions (opt-in via SKILLOPT_UNSAFE=1) - Include raw output in JSON for smoke test evidence - Fix smoke script: fail on errors, preserve raw output
|
Thanks—the latest commit fixes pipe-separated alternatives, adds false-completion regressions, scrubs the inherited environment, makes unsafe mode opt-in, and preserves raw output. Those are meaningful improvements. A few blockers remain:
Once these are addressed, we can re-review promptly. |
- Add file_exists judge op for external execution evidence (not stdout parsing) - Update flaky scenario to require .test_passed sentinel (proves rerun) - Fail explicitly on missing candidate path (FileNotFoundError) - CLI exits non-zero when any scenario has error - Use --allowedTools by default instead of blanket permission bypass - Symlink auth from real HOME to preserve Claude login in isolated env - Add SECURITY.md documenting execution model and limitations - Add smoke test artifacts as merge evidence (score: 1.0) Regression tests added: - test_file_exists_positive/negative - test_false_self_report_regression - test_flaky_no_rerun_regression - test_nonexistent_candidate_raises - test_default_uses_scoped_permissions - test_unsafe_mode_uses_permission_bypass Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new SuperpowersEvaluator adapter to run synthetic, offline-evaluable scenarios against a pinned Superpowers checkout, along with a rule-based judge, tests, and a manual smoke script/security doc to validate real-harness runs.
Changes:
- Added
skillopt_sleep/adapters/superpowers.pyimplementing scenario setup, Superpowers checkout/overlay, Claude execution, and rule-based scoring. - Added
tests/test_superpowers_scenarios.pycovering judge ops and overlay/CLI fail-closed behaviors with mocked subprocess execution. - Added a manual smoke script + security notes, and committed sample smoke outputs under
smoke_results/.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
skillopt_sleep/adapters/superpowers.py |
Core adapter: embedded scenarios, judge implementation, temp workspace setup, Superpowers clone + skill overlay, scenario execution and scoring |
skillopt_sleep/adapters/__init__.py |
Initializes adapters package |
tests/test_superpowers_scenarios.py |
Offline tests for judge ops and overlay mechanics (mocked subprocess.run) |
scripts/smoke_superpowers.sh |
Manual smoke runner that writes JSON outputs |
docs/superpowers/SECURITY.md |
Documents security model/limitations for running untrusted candidate skills |
.gitignore |
Ignores uv.lock |
smoke_results/20260721_215806/baseline.json |
Committed smoke artifact output |
smoke_results/20260721_220108/baseline.json |
Committed smoke artifact output |
smoke_results/20260721_220638/baseline.json |
Committed smoke artifact output |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| pos1 = output.lower().find(args[0].lower()) | ||
| pos2 = -1 | ||
| for pat in args[1].split("|"): | ||
| p = output.lower().find(pat.lower()) | ||
| if p >= 0: | ||
| pos2 = p | ||
| break | ||
| return pos1 >= 0 and pos2 >= 0 and pos1 < pos2 |
| sid = scenario["id"] | ||
| scenario_seed = random.randint(0, 2**31 - 1) | ||
| result = ScenarioResult( | ||
| id=sid, passed=False, | ||
| pinned_sha=pinned_sha, candidate_hash=candidate_hash, scenario_seed=scenario_seed, | ||
| ) |
| # Symlink auth-related files from real HOME (credentials, not config) | ||
| real_claude_dir = Path.home() / ".claude" | ||
| if real_claude_dir.exists(): | ||
| for auth_file in ["credentials.json", ".credentials.json", "settings.json"]: | ||
| src = real_claude_dir / auth_file | ||
| if src.exists(): | ||
| dst = claude_dir / auth_file | ||
| if not dst.exists(): | ||
| dst.symlink_to(src) | ||
|
|
| cmd = ["claude", "-p", prompt] | ||
|
|
||
| if os.environ.get("SKILLOPT_UNSAFE") == "1": | ||
| import warnings | ||
| warnings.warn( | ||
| "SKILLOPT_UNSAFE=1: Running with --dangerously-skip-permissions. " | ||
| "Do not use with untrusted candidate skills.", | ||
| stacklevel=2, | ||
| ) | ||
| cmd.append("--dangerously-skip-permissions") | ||
| else: | ||
| # Scoped permissions: allow only tools needed for test scenarios | ||
| # Bash for pytest, Edit/Write for fixing code, Read for inspection | ||
| cmd.extend([ | ||
| "--allowedTools", "Bash,Edit,Write,Read", | ||
| ]) | ||
|
|
||
| t0 = time.time() | ||
| try: | ||
| proc = subprocess.run( | ||
| cmd, | ||
| cwd=str(project_dir), | ||
| capture_output=True, | ||
| text=True, | ||
| timeout=timeout, |
| skill: str | ||
| version: str | ||
| scenarios: List[ScenarioResult] = field(default_factory=list) |
| return { | ||
| "skill": self.skill, | ||
| "version": self.version, | ||
| "score": self.score, |
| Raises: | ||
| FileNotFoundError: if candidate_skill_path is provided but doesn't exist | ||
| """ | ||
| results = EvalResults(skill=self.skill, version=self.version) |
| # Usage: | ||
| # SKILLOPT_UNSAFE=1 ./scripts/smoke_superpowers.sh [candidate_skill_path] | ||
| # | ||
| # Output: writes results + raw output to smoke_results/ for PR evidence. | ||
| # Fails on any runner error (no silent swallowing). |
| echo "Results saved to $OUTDIR" | ||
| echo "Include these files in your PR as evidence of smoke test." |
| 1. **Scrubbed environment**: Only essential vars passed (HOME, PATH, TERM, LANG, ANTHROPIC_API_KEY) | ||
| 2. **Isolated HOME**: Each scenario gets its own HOME directory | ||
| 3. **No `--dangerously-skip-permissions` by default**: Permission prompts required unless explicitly bypassed | ||
| 4. **Project directory isolation**: Each scenario gets its own project directory |
|
Thanks for the latest update. We re-ran head A few merge blockers remain:
Please also address the related Copilot inline findings—especially deterministic provenance/top-level pinned SHA, |
…isolation Addresses remaining maintainer + Copilot review blockers on microsoft#134. - Load the pinned checkout via the normal plugin bootstrap (`claude --plugin-dir`), not a hand-rolled skills symlink. A per-run session marker is injected into using-superpowers/SKILL.md and required in the agent's output, proving the SessionStart/using-superpowers activation actually ran. - Replace agent-writable sentinel files with harness-owned evidence: a pytest/python shim on PATH logs every invocation outside the project dir, and the harness re-runs pytest itself after the agent exits. Scenarios now score pytest_runs and harness_test_passes; forged files no longer satisfy any check. - Stop reusing host credentials by default. ~/.claude auth/settings are no longer symlinked; reuse is opt-in via SKILLOPT_HOST_AUTH=1 (warns). Fail closed (NO_AUTH) when neither a key nor host-auth is available. - Add OS-level isolation, opt-in via SKILLOPT_SANDBOX=bwrap|docker. - Prompt on stdin + --output-format text, matching backend.py CLI usage. - Deterministic scenario seed (SHA + id), pinned_sha carried on EvalResults and in to_dict(); order op accepts any alternative occurring after the first token. - Stop committing smoke_results/ (raw output + host paths); smoke script now writes gitignored raw JSON plus sanitized *.summary.txt excerpts to share. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Thanks for the thorough passes. Pushed 1. Real baseline-vs-candidate smoke, non-empty candidate hashBoth arms run the pinned SHA
Same pinned SHA / model / scenario for both. Candidate hash is non-empty, authentication is real (host OAuth via opt-in 2. Evidence is no longer forgeable by the agentSentinel files in the agent-writable project are gone. Evidence is now harness-owned:
Regression tests confirm forged 3. Real execution boundary
Copilot inline findings
Tests: 44 focused pass; full suite |
| project_dir = workspace / f"project-{sid}" | ||
| project_dir.mkdir(parents=True, exist_ok=True) | ||
| scenario_home = workspace / f"home-{sid}" | ||
| scenario_home.mkdir(parents=True, exist_ok=True) | ||
| audit_dir = workspace / f"audit-{sid}" | ||
| audit_dir.mkdir(parents=True, exist_ok=True) | ||
| audit_log = audit_dir / "pytest.log" | ||
| bin_dir = workspace / f"bin-{sid}" | ||
| _write_pytest_shims(bin_dir, audit_log) |
| bootstrap_skill = superpowers_dir / "skills" / "using-superpowers" / "SKILL.md" | ||
| if bootstrap_skill.exists(): | ||
| bootstrap_skill.write_text( | ||
| bootstrap_skill.read_text() | ||
| + f"\n\n## Session marker\n\nEnd your final message with the line `{marker}`.\n" | ||
| ) |
| return [ | ||
| "docker", "run", "--rm", "-i", | ||
| "-v", f"{project_dir}:{project_dir}", | ||
| "-v", f"{home}:{home}", | ||
| "-v", f"{plugin_dir}:{plugin_dir}:ro", | ||
| "-w", str(project_dir), | ||
| "-e", "HOME", "-e", "ANTHROPIC_API_KEY", "-e", "SKILLOPT_ATTEMPT", | ||
| image, | ||
| ] |
…checkout reuse Addresses 3 Copilot inline findings on 674d1db: - Move the pytest shim + audit log under scenario HOME (was under the bare workspace). bwrap/docker only mount project_dir, HOME and plugin_dir, so the old location left the shim invisible inside the sandbox and pytest_runs stuck at 0. - Pass PATH/LANG/TERM into the docker sandbox so the shimmed PATH carries into the container; without PATH the shim dir dropped off and invocations weren't counted. - Strip any prior "## Session marker" block before injecting the current one. The checkout is reused across scenarios, so appending accumulated markers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.
Comments suppressed due to low confidence (2)
skillopt_sleep/adapters/superpowers.py:614
superpowers_versionis treated as user-facing configuration, but it’s not used to select what gets checked out (the clone is pinned solely bypinned_sha). This is easy for callers to misinterpret as “I’m evaluating tag vX.Y.Z”, when they’re actually evaluating whatever SHA is passed.
def __init__(
self,
skill: str = "verification-before-completion",
superpowers_version: str = DEFAULT_VERSION,
timeout: int = DEFAULT_TIMEOUT,
token_cap: int = 0,
):
self.skill = skill
self.version = superpowers_version
self.timeout = timeout
self.token_cap = token_cap # 0 = no cap
docs/superpowers/SECURITY.md:20
- This mitigation text says the scenario
HOMEis "empty", but the adapter creates an isolated HOME tree (e.g.,.claude/and.skillopt/) for shims/logs. Rewording avoids implying HOME has no files, only that no host Claude config is reused.
1. **No host credential reuse by default.** The scenario `HOME` is empty; host
`~/.claude/credentials.json` and `settings.json` are never copied or
symlinked. Reuse is opt-in via `SKILLOPT_HOST_AUTH=1`, which warns.
| # Isolated project, HOME and (harness-only) audit dir per scenario | ||
| project_dir = workspace / f"project-{sid}" | ||
| project_dir.mkdir(parents=True, exist_ok=True) | ||
| scenario_home = workspace / f"home-{sid}" | ||
| scenario_home.mkdir(parents=True, exist_ok=True) | ||
| # shim + audit log live under HOME so they're visible inside the sandbox | ||
| # (bwrap/docker mount HOME but not the bare workspace) | ||
| audit_log = scenario_home / ".skillopt" / "pytest.log" | ||
| bin_dir = scenario_home / ".skillopt" / "bin" | ||
| _write_pytest_shims(bin_dir, audit_log) |
| claude_dir = scenario_home / ".claude" | ||
| claude_dir.mkdir(parents=True, exist_ok=True) |
| def _write_pytest_shims(bin_dir: Path, audit_log: Path) -> None: | ||
| """Install harness-owned `pytest`/`python` shims that log real invocations. | ||
|
|
||
| The log lives outside the agent's project directory and the shims always | ||
| exec the real interpreter, so an invocation can be counted but not forged | ||
| from inside the project. Same pattern as the tool shims in backend.py. | ||
| """ | ||
| bin_dir.mkdir(parents=True, exist_ok=True) | ||
| real_python = sys.executable | ||
|
|
||
| def _install(name: str, body: str) -> None: | ||
| path = bin_dir / name | ||
| path.write_text(f"#!/usr/bin/env bash\n{body}") | ||
| path.chmod(path.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) |
| for scenario in scenarios: | ||
| if scenario_filter and scenario["id"] != scenario_filter: | ||
| continue |
| # No || true - fail if runner errors | ||
| python -m skillopt_sleep.adapters.superpowers "${args[@]}" > "$outfile" | ||
|
|
||
| # Sanitized summary for sharing: no raw output, no host paths, no secrets |
Summary
Adds adapter to evaluate Superpowers skills against synthetic scenarios. Refs #132.
Changes
skillopt_sleep/adapters/superpowers.py:SuperpowersEvaluatorclassverification-before-completionskillskills/<name>/SKILL.md, symlinksHOME/.claude/skillstests/test_superpowers_scenarios.py: 20 testsscripts/smoke_superpowers.sh: Manual baseline vs candidate comparisonUsage
Or CLI:
Scenarios
test-passes-verifytest-fails-no-claimpremature-claim-resistpartial-pass-honestflaky-verify-rerunTest plan
pytest tests/test_superpowers_scenarios.py)scripts/smoke_superpowers.sh(opt-in, not CI)