feat(evals): add completion-rate baseline harness - #1032
Merged
TheGreatAxios merged 4 commits intoSep 15, 2026
Merged
TheGreatAxios merged 4 commits into
TheGreatAxios merged 4 commits into
Conversation
Summary: repeatable harness runs a frozen 4-task set end to end through the production agent loop with scripted mock responders, reporting completion rate plus control-layer signals (turns, retries, compaction events, doom-loop/thrash interventions, wall clock). Baseline on b509ed3: 50.0% (4/8 runs, 2 repeats), deterministic across repeats. Task set frozen as tasks.json v1 for the post-0.4.x re-measure. Verification: bun scripts/eval-completion.ts --repeats 2 --out evals/completion/baseline-2026-09-14.json; bun test evals/completion/lib.test.ts (7 pass); bun run check (exit 0).
…dline The sync finally cleared the timer before Promise.race settled, so RunStatus timeout was unreachable and --timeout-ms was a no-op. Await the race so the timer clears on settle, export withTimeout for testing, and guard main() with import.meta.main. Adds scripts/eval-completion.test.ts proving a hung run rejects after the timeout. Baseline re-run still 50.0% (4/8).
TheGreatAxios
force-pushed
the
cl-7932-baseline-task-completion-rate-before-the-04x-line-and-re
branch
from
September 14, 2026 17:54
5983dc2 to
77b8c30
Compare
…e suspensions (#1047) A run that times out after emitting a failure signal kept the timeout status instead of being recorded failed. The deadline now aborts the in-flight send and the runner quiesces the agent pump and collector before grading, so verify never runs on partial state. Gate suspensions are summed into completion totals and printed in the human summary instead of being collected and dropped.
#1049) * fix(evals): contain completion-harness task paths and validate timeout Confine custom task-set fixture/script/verify references to the harness directory, mark the version-controlled grading trust boundary, reject degenerate timeout values, and re-record the baseline against the current tree. * fix(evals): align baseline and totals with gate-suspension signal Carry the gate-suspension total through schema, aggregation, and summary so the checked-in baseline validates. Add a regression test asserting the frozen baseline against the current report schema. * fix(evals): label estimated turns and failed tool calls honestly (#1062)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
bun run checkpasses (lint, typecheck, build, guarded suite)Fixes CL-7932