Three questions a provider must answer, drawn from eight that were run - #43
Conversation
The last of the provider research, and written last on purpose: the instruction was to define the abstraction only after the prototypes revealed the common minimum, and not from tool names. What is in ADR 0005 comes from eight evaluations that were actually run -- cargo-deny, zizmor, cargo-mutants, cargo-fuzz, ast-grep, semgrep, CodeQL, kani. Two adopted, one documented as a manual tier, five not adopted. What they have in common is three questions, not an interface: 1. **Is there a name this repository can read?** The split that matters is not the tool, it is whether its configuration is local. A `deny.toml`, an ast-grep rule directory, a `zizmor.yml`, a `repo: local` hook and a lefthook command are all files in this tree, so "is this id defined here" is answerable. A hook pinned at a rev elsewhere is not, and what can be verified there is the pin. Neither class ever says what an id MEANS. 2. **Has the refusal been seen?** `uphold probe` was written for hooks and the property it establishes is not about hooks. The evaluations produced two gates that could not refuse without anybody noticing: a semgrep rule whose sanitizer sat after its sink, and a CodeQL query with no model for the builder API. Both ran. Both reported. Neither reported the thing it was written for. 3. **Can it say "I could not look", and does it?** ast-grep exits 0 over a file whose parse collapsed. CodeQL prints "Successfully created database" and returns zero rows, with the parse error four rows into a channel already carrying 1,689 diagnostics at the same severity. A pin check counted an unreachable remote as resolved, here, before the guard was fixed. The pattern does not weaken as the tier gets more expensive; it gets harder to notice, because a 600 MB toolchain printing "successfully" is more convincing than a regex printing nothing. So a provider that cannot tell those apart must be PAIRED with something that can, and the pairing belongs in the claim rather than in the adopter's memory. What the contract is not: a plugin API, an embedding, or a rule DSL. Eight evaluations asked for none of them. The providers share three questions and no input format, no output format, no configuration shape and no execution model. The cost table is measured on this tree, so the ratios are the point: the content scan is milliseconds and `cargo mutants` over one module is eight minutes. A claim naming a rule must also name where it runs, because "enforced here" is false when the seam is one nobody can afford to trigger -- which is the decision this repository has already made three times in the same direction. ROADMAP records the half of the upstream-verification limit that is closable, which is the one feature this whole exercise argues for: ask a provider whose config is local whether the claimed id is defined in it.
|
Warning Review limit reached
Next review available in: 14 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #43 +/- ##
=======================================
Coverage 89.44% 89.44%
=======================================
Files 31 31
Lines 9732 9732
=======================================
Hits 8705 8705
Misses 1027 1027 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
2 participants
The last deliverable of #13: define the generic provider contract, and only after the prototypes revealed the common minimum.
Written last on purpose. What is in ADR 0005 comes from eight evaluations that were run --
cargo-deny,zizmor,cargo-mutants,cargo-fuzz,ast-grep,semgrep, CodeQL,kani-- rather than from a survey of tool names. Two adopted, one documented as a manual tier, five not adopted.The contract is three questions, not an interface
1. Is there a name this repository can read? The split that matters is not the tool, it is whether its configuration is local. A
deny.toml, an ast-grep rule directory, azizmor.yml, arepo: localhook and a lefthook command are all files in this tree, so "is this id defined here" is answerable -- and that is a much smaller feature than a rule DSL. A hook pinned at a rev elsewhere is not; what can be verified there is the pin. Neither class ever says what an id means.2. Has the refusal been seen?
uphold probewas written for hooks, and the property it establishes is not about hooks. The evaluations produced two gates that could not refuse without anybody noticing: a semgrep rule whose sanitizer sat after its sink, and a CodeQL query with no model for the builder API. Both ran, both reported, neither reported the thing it was written for.3. Can it say "I could not look", and does it?
ast-grepSuccessfully created database, zero rows; theparse_erroris four rows in a channel already carrying 1,689 diagnostics at the same severityThe pattern does not weaken as the tier gets more expensive. It gets harder to notice, because a 600 MB toolchain printing "successfully" is more convincing than a regex printing nothing. So a provider that cannot tell those apart must be paired with something that can, and the pairing belongs in the claim rather than in the adopter's memory.
What it is not
Not a plugin API, not an embedding, not a rule DSL over tree-sitter. Eight evaluations asked for none of them: the providers share three questions and no input format, no output format, no configuration shape and no execution model.
Cost decides the seam
The table is measured on this tree, so the ratios are the point -- the content scan is milliseconds,
cargo mutantsover one module is eight minutes. A claim naming a rule must also name where it runs, because "enforced here" is false when the seam is one nobody can afford to trigger. This repository has made that call three times already, in the same direction each time.ROADMAP.mdnow records the half of the upstream-verification limit that is closable, which is the one feature this whole exercise argues for.Documentation only -- no code changes.