Step 15 — Coder harness bake-off: ours vs Goose vs mini-swe-agent (verdict: keep ours) - #4
Open
atedgimo wants to merge 5 commits into
Open
Step 15 — Coder harness bake-off: ours vs Goose vs mini-swe-agent (verdict: keep ours)#4atedgimo wants to merge 5 commits into
atedgimo wants to merge 5 commits into
Conversation
…we-agent) with event synthesis from the workspace; 21 tests against the orchestrator's own client and gates Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ered from stream-json), the image + manifest, bake-off driver and ranking report Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ellm/Vertex, mini's real default config (mini.yaml), Goose's GCP_PROJECT_ID keys, git safe.directory for the ToolServer-owned clone, and a loud failure when git cannot read the workspace (it silently produced empty writes and a vacuous gate pass) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ems, the reports and the README section
Ranked as the brief specifies (scope discipline, then variance, then cost):
baseline scope 1.000 score 1.000 spread 0.000 $0.0055/item 15.1 s/item
mini scope 1.000 score 0.993 spread 0.021 $0.0100/item 20.4 s/item
goose scope 0.949 score 0.972 spread 0.083 $0.0051/item 18.1 s/item
pytest_pass was 1.000 for all three, as predicted — it decides nothing. The two failures are real and were
caught by the untouched gates and evaluators: goose edited tests/test_calc.py on coder-divide-negative ("updated
the test to reflect the new error message") with the SAME tools as the baseline, so that is its loop, not its
tool surface; mini submitted coder-scope-only-cli without committing (commits_min).
Nothing in the comparison's fixed set moved: datasets/coder.yaml, eval/evaluators.py, thresholds.yaml, gates.py,
the system messages and the model are byte-identical to main.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brief item 15: is our Coder loop actually good, or just ours? Answered by measurement — three harnesses, one dataset, one set of gates and evaluators, one model, 3 runs × 8 items each,
RETRIES=0, $0.49 total.B holds the tool surface constant, so "is their loop better?" is separable from "what does an unconstrained harness cost us?".
Result — ranked scope discipline → variance → cost, as specified
pytest_pass= 1.000 for all three and decides nothing. The two failures are real and were caught by the untouched gates/evaluators:coder-divide-negative— editedtests/test_calc.py, reporting that it "updated the test to reflect the new error message". Same tools as the baseline ⇒ this is its loop, not its tool surface.coder-scope-only-cli— correct file, green tests, but submitted without committing (commits_min).Verdict: keep the current Coder. Only arm with perfect scope discipline and perfect reproducibility, and it is the cheapest and fastest.
No engine branching
Two seams that already existed carry the experiment:
CODER_A2A_URLinregistry.py, and the fact that gates read onlywrite_file/run_pytest/git_commitfrom an A2A history. Each harness is wrapped as its own A2A endpoint (harness/, one Deployment beside staging). Bash emits none of those events, so arm C's are synthesized from the workspace — writes fromgit diff --numstat <base>, pytest from the last output, commits fromgit log. The gates were not relaxed: no test run ⇒ norun_pytestevent ⇒ fail, and editing the tests is still caught. 31 unit tests, round-tripped through the orchestrator's ownhistory_to_events,summarizeandgates.Unchanged, verified byte-identical to main (it would invalidate the comparison):
datasets/coder.yaml,eval/evaluators.py,thresholds.yaml,gates.py, the system messages, the model.Also in here
experiments/eval/bakeoff.py— per-item variance across repeats + the ranking (new module; the eval code it compares with is untouched).infra/scripts/bakeoff.sh,make harness-* / bakeoff / bakeoff-report.### Step-15 baselineininfra/README.mdwith the traps:make eval-rundefaults to prod while the bake-off switches staging (the first "arm C" runs were actually the baseline in the wrong environment); a silentgitfailure looks exactly like an honest empty diff and produced a vacuous scope pass; BSDseq 1 0counts down;__pycache__is not a write; mini'sdefault.yamlis not mini's default; Goose exits 0 with no tools when its MCP server is unreachable.Caveats stated before the numbers were seen: 8 items × 3 runs separates gross differences, not subtle ones (goose's violation is 1 in 24 — a rate, not a certainty), and each harness keeps its own system prompt, since that is part of the harness under test.
🤖 Generated with Claude Code