feat: a script re-prepares a pull request with another model and compares the findings with the bundle it has - #100
Merged
Merged
Conversation
…ares the findings with the bundle it has `scripts/inbox-compare.ts <owner/repo#n> [--model m] [--effort e]` takes the newest bundle the inbox has for a pull request as the baseline, prepares the same head again with the candidate settings, and prints a markdown table of what was reproduced, what is new, and what the run spent. The candidate gets its own scratch worktree and diffity data directory under a temp directory, so nothing is written into ~/.diffity and a running daemon is undisturbed. A worktree can only be cut at whatever refs/pull/<n>/head points at, so the script asks for that head before spending an agent run and refuses when the pull request has moved past the baseline. The scripts directory is now typechecked too, which is what keeps this one honest. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bwp5QefjsjMFeq6CK6cT6w
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bwp5QefjsjMFeq6CK6cT6w
A worktree is not limited to whatever refs/pull/<n>/head points at now: an earlier head of the pull request is usually an ancestor of the current one, and the forge serves any commit reachable from an advertised ref by sha. `prepareWorktree` takes an optional pinHead and checks that commit out — present already, fetched by sha, or refused as force-pushed away — and `preparePr` passes it through from PrepareOpts. The daemon sets neither and behaves exactly as before. The compare script pins the baseline's head and drops the pre-flight refusal, so the six bundles on this machine whose pull requests have moved on are comparable again — including #14550, one of the two P1-in-a-tiny-diff cases the adoption gate needs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bwp5QefjsjMFeq6CK6cT6w
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/inbox-compare.ts <owner/repo#n> [--head sha] [--model m] [--effort e]measures acandidate drafter against what the inbox already found, so
agent.modelis decided on findingsrather than on price.
It takes the newest bundle for the pull request under
~/.diffity/inbox/bundlesas the baseline,prepares the same head again through
preparePrwith the candidate settings (no filter, novalidate pass, no MCP), and prints a markdown table: per severity how many baseline findings were
reproduced and how many candidate findings are new, plus the run's cost, minutes, turns and output
tokens. A finding counts as reproduced on the same file with an overlapping line range, a one-line
finding taken as its line ±2.
--jsonprints the same numbers;--outwrites the markdown.The worktree and every diffity data directory the run uses live under a fresh temp directory
(
--scratchto name it), so nothing is written into~/.diffityand a running daemon isundisturbed. A worktree can only be cut at whatever
refs/pull/<n>/headpoints at, so the scriptasks for that head first and refuses — before spending an agent run — when the pull request has
moved past the baseline. Exit 0 on a comparison, 2 on a candidate that skipped or failed, 1 on a
usage error.
development.mdgains a section on reading the table, including the caveat that the baselinebundles were prepared by the old pipeline (reviewer's settings, skills and MCP loaded, toolchain
runs allowed), so a difference is prompt and model.
Verified: 34 new unit tests over the pure parts (argument parsing, newest-bundle selection, the
±2 matching and the tallies, general and dismissed threads excluded, the markdown and the JSON
shape); the whole
scripts/directory is now typechecked, which is wired intonpm run typecheck;full
npm testgreen. The script's pre-agent paths were exercised against a throwaway local clone— usage errors, missing clone, missing bundle, and the moved-head refusal — and the rendering
against the real 14550 and 14493 bundles. The comparison itself has not been run: each run spends
real agent time, and that is the reviewer's call after this merges.
Part of #93
🤖 Generated with Claude Code
https://claude.ai/code/session_01Bwp5QefjsjMFeq6CK6cT6w