Review Engine for Evaluating Versioned Edits
A GitHub App that reviews pull requests with an LLM and publishes line-anchored comments.
REEVE listens for pull request events, checks the code out into an isolated worktree, and runs four specialised reviewers over the diff — security, performance, bugs, and code quality.
Findings are grounded in static analysis, deduplicated per line, ranked, and published as a single review with a summary and inline comments.
webhook -> queue -> worker -> review -> GitHub
202 immediately everything else in the background
uv sync
cp .env.example .env # fill in the four required settings
uv run pytest # 388 tests, ~4s
uv run uvicorn pr_review_agent.main:app --reloadOr with Docker:
docker compose up -dThe webhook needs a publicly reachable URL — see Local Development.
| Stage | What happens |
|---|---|
| Ingest | Webhook verified by HMAC, event filtered, job queued, 202 returned |
| Check out | Installation token, cached bare mirror, detached worktree |
| Diff | From the last reviewed commit, or the base on a first pass |
| Filter | Docs, lockfiles and vendored code never reach a reviewer |
| Ground | ruff runs first, so reviewers see what tools already found |
| Review | 4 reviewers × N files, bounded concurrency |
| Narrow | Off-diff lines dropped, duplicates collapsed, findings capped |
| Publish | One review: summary body plus inline comments |
Comments are anchored, not guessed. The diff handed to the model is pre-numbered, and anything landing outside the diff is dropped locally. A single out-of-range line makes GitHub reject an entire review.
Agreement is a signal. Four reviewers flagging one line becomes one comment recording who agreed. Corroboration then outranks self-reported confidence when the cap chooses.
Failure is visible. A rejected review degrades to a summary comment. A timed-out one propagates rather than risking a duplicate post. Queued reviews are drained on shutdown.
Quality is measured.
eval/ scores the real reviewers against fixtures carrying both known defects
and known-clean lines.
Measured on a generated 18-file pull request with 18 planted defects:
| Metric | Result |
|---|---|
| Defective modules found | 6 of 6 (both runs) |
| Model calls | 72, zero failures |
| Wall clock | ~44s |
| Published inline | 10 of 30 findings |
| Clean modules drawing comments | 5 of 12 |
Recall is strong. Precision on clean code is not, and a fixed comment cap is the wrong instrument for a large diff — two thirds of findings never appear inline. Both are open problems.
src/pr_review_agent/
├── api/ webhook ingress
├── github/ auth, client, review publishing
├── services/ git worktrees, review workflow, review store
├── parsers/ unified diff → PullRequestFile
├── ai/ prompts, reviewers, orchestrator, dedup,
│ cap, summary, static analysis
├── jobs/ queue, worker
├── dependency.py composition root
└── models.py frozen dataclasses
eval/ precision / recall harness
Adding a review concern takes exactly three things:
- A
Promptimplementation inai/prompts/ - Tests in
tests/ai/ - A
PromptedReviewerregistration independency.py
It must not touch ReviewService, AIReviewEngine, ReviewOrchestrator, the
provider, the GitHub clients, or GitService. That rule has survived every
reviewer added and removed so far.
| Architecture | Boundaries, and why they sit where they do |
| Review Pipeline | Diff to published comment |
| Configuration | Every setting and its rationale |
| Evaluation | The harness, and how to read it |
| Local Development | Running it yourself |
| Deployment | Docker, volumes, shutdown |
| CI | The pipeline |
| Changelog | What shipped, and what has not |
- Reviewers see one file's diff — no cross-file or repository context. This is the main ceiling on review quality.
- Static analysis covers Python only.
- Review state is in memory; a restart costs one redundant review per open PR.
- Prompt injection is mitigated by wording, not by a boundary.
- A crash still loses queued reviews.
FastAPI · httpx · GitPython · pydantic-settings · ruff · uv · pytest