Skip to content

fix(test-writing): degrade malformed defense stances and add deterministic evidence gates - #177

Merged
Martin Bens (SpiGAndromeda) merged 4 commits into
mainfrom
fix/team-review-defense-crash-and-evidence-gates
Sep 5, 2026
Merged

Martin Bens (SpiGAndromeda) merged 4 commits into
mainfrom
fix/team-review-defense-crash-and-evidence-gates

Conversation

@SpiGAndromeda

@SpiGAndromeda Martin Bens (SpiGAndromeda) commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

The phpunit-test-team-reviewing pipeline survives a malformed defense stance instead of discarding the run, and stops trusting two agent-reported values it can check itself. test-writing goes 5.1.0 to 5.1.1. Closes #173; implements the manifest-count, finding-evidence, fix-application, commit-reachability, scrutiny-label, and re-characterization items of #176 — the structure-only layout consolidation item is not part of this change.

Defense stage degrades instead of aborting

A mode=adversarial run threw at the wave boundary when a defense reconciler listed an adversary-introduced finding under both adopted_new and findings: the maintained-findings fold resolved the id against the file's kept and contested records only, found nothing, and the throw discarded every completed agent in the run. The duplicated entry is now skipped as an already-cast vote (seenA/seenR are per-defender, so they answer exactly whether this defender already voted on the id), and defensePrompt states that the four response arrays are disjoint, removing the ambiguity that produced the duplication in the first place.

Beyond that fast path, any defense stance entry failing an integrity guard — a malformed finding_id, or an id resolving to no known record — takes the degrade-by-role path from references/error-handling.md that already covered a dead defense reconciler: the offending entry is dropped, the rest of that stance still votes, the finding it named keeps its prior consensus binding, and each drop is recorded in red_team.defense_degraded (same null-when-clean shape as coverage_gap) and rendered as a CAUTION block, so a degraded defense wave is never presented as a complete one. A dropped promotion is credited in no metric.

Re-characterizations survive the merge

A defender can maintain a finding while correcting its location, method, and remediation after re-reading the file. The merge picked the descriptive owner by remediation length, so a terser correction lost every corrected field back to the stale original and its fix landed behind the original's in suggested_variants — a fix applier working from the report would change code that was never defective. Maintained findings now merge through recharacterize(), which makes the defender's payload the descriptive owner whenever it proposed a remediation. A payload without one keeps the original's fields, preserving the invariant that current and suggested describe one change.

Deterministic gates on agent-reported data

Two values flowed through the pipeline unvalidated although a script can check both. The gates run skill-side between stages, because the workflow script is filesystem-sandboxed and carrying file bodies in the manifest would breach the run-script size cap.

Gate Point Effect
workflow/verify-method-counts.sh Phase 1, before the manifest freezes Re-extracts every entry's test method names from disk; a mismatched method_count/test_methods is replaced with the extracted truth and logged, never merely warned
workflow/verify-finding-evidence.sh Phase 5, before the merge A kept finding whose non-empty current block does not occur in the target file under whitespace normalization is demoted to contested with an outcome naming the failed match; findings that quoted no code are exempt

The evidence gate also syncs each demotion into the file's adversarial_input, so a fabricated quote never reaches the merge, the red team, or the report as kept — one invocation covers Phase 5 and Phase 6. Both scripts validate their input shape per entry and fail hard on a corrupted manifest or result instead of repairing it.

Scrutiny labels and the fix-application contract

The report never distinguished findings that survived a red team from findings nothing ever challenged. Every rendered finding now carries Scrutiny: adversary-tested | consensus-only, derived from which stage produced the file's final per-finding state — never asserted per finding by any agent.

The new references/fix-application.md is the contract for whoever applies a report's remediations: apply the report's final suggested verbatim and keep suggested_variants as human-facing alternatives; self-review the fix diff before committing (DESIGN-003/DESIGN-004 across co-touched files, a tautology check on every added test, and the same PHPStan/php-cs-fixer gates the reviewed code passes through); judge any mutant against the whole suite, never a single test; re-verify the factual premise of a consensus-only must-fix finding before applying it; and verify every reported commit with git merge-base --is-ancestor before calling it landed.

The adversarial stage threw at the wave boundary when a defense reconciler listed an adopted finding under both `adopted_new` and `findings`, discarding every completed agent in the run. The duplicate is now skipped as an already-cast vote, and `defensePrompt` states explicitly that the four response arrays are disjoint, removing the ambiguity that produced the duplication.

Beyond that fast path, any defense stance entry that fails an integrity guard now takes the degrade-by-role path from `references/error-handling.md` instead of throwing. The offending entry is dropped, the finding it named keeps its prior consensus binding, and each drop is recorded in `red_team.defense_degraded` so an incomplete defense wave is never rendered as a complete one.

The defense fold also lost re-characterizations. `mergeRemediations` picks the descriptive owner by remediation length, so a defender correcting a finding's location, method, and remediation with a terser fix lost every corrected field back to the stale original. Maintained findings now merge through `recharacterize`, which makes the defender's payload the descriptive owner whenever it proposed a remediation. A payload without one keeps the original's fields so `current` and `suggested` still describe one change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The team-review pipeline trusted two agent-reported values it can verify itself. Extraction subagents' `method_count` and `test_methods` drove the track decision and shard scoping unchecked, so one wrong count silently misrouted a file. `verify-method-counts.sh` now re-extracts the method names from disk before the manifest freezes, replaces mismatches with the extracted truth, logs each replacement, and fails hard on a corrupted entry instead of repairing it.

A reviewer finding's `current` block was adjudicated by consensus, the red team, and the report without anyone checking the quoted code exists. `verify-finding-evidence.sh` runs over each persisted shard result before the merge. A kept finding whose non-empty `current` does not occur in the target file under whitespace normalization moves to `contested` with an outcome naming the failed match, and the demotion is synced into `adversarial_input`, so a fabricated quote never reaches the merge, the red team, or the report as kept. A finding that quoted no code is exempt.

The rendered report now labels every finding with the scrutiny it actually received, `adversary-tested` or `consensus-only`, derived from which stage produced the finding's final state. The new `fix-application.md` reference carries the contract for whoever applies a report's remediations: apply `suggested` verbatim, self-review the fix diff for redundancy and tautological tests plus the static gates, judge mutants against the whole suite only, re-verify the premise of a `consensus-only` must-fix finding before applying it, and confirm reported commits with `git merge-base --is-ancestor` before calling them landed.

Both gate scripts validate their input shape per entry and are covered by 18 BATS tests in plugin-tests/test-writing, wired into the skill's Phase 1 and Phase 5 with matching digraph nodes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Patch release carrying the adversarial-stage degradation fixes and the deterministic manifest and evidence gates; CHANGELOG entry derived against main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

… tests

Three fixtures quote PHP code containing `$x` and `$doesNotExist` in single quotes on purpose, and CI's shellcheck run over `.bats` files fails on the resulting SC2016 notes. Each site gets a per-line disable directive with the reason, matching the convention in `plugin-tests/mcp-shared/environment.bats`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adversarial stage aborts the whole run when a defense reconciler repeats an adopted finding under findings

1 participant