fix(test-writing): degrade malformed defense stances and add deterministic evidence gates - #177
Merged
Martin Bens (SpiGAndromeda) merged 4 commits intoSep 5, 2026
Conversation
The adversarial stage threw at the wave boundary when a defense reconciler listed an adopted finding under both `adopted_new` and `findings`, discarding every completed agent in the run. The duplicate is now skipped as an already-cast vote, and `defensePrompt` states explicitly that the four response arrays are disjoint, removing the ambiguity that produced the duplication. Beyond that fast path, any defense stance entry that fails an integrity guard now takes the degrade-by-role path from `references/error-handling.md` instead of throwing. The offending entry is dropped, the finding it named keeps its prior consensus binding, and each drop is recorded in `red_team.defense_degraded` so an incomplete defense wave is never rendered as a complete one. The defense fold also lost re-characterizations. `mergeRemediations` picks the descriptive owner by remediation length, so a defender correcting a finding's location, method, and remediation with a terser fix lost every corrected field back to the stale original. Maintained findings now merge through `recharacterize`, which makes the defender's payload the descriptive owner whenever it proposed a remediation. A payload without one keeps the original's fields so `current` and `suggested` still describe one change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The team-review pipeline trusted two agent-reported values it can verify itself. Extraction subagents' `method_count` and `test_methods` drove the track decision and shard scoping unchecked, so one wrong count silently misrouted a file. `verify-method-counts.sh` now re-extracts the method names from disk before the manifest freezes, replaces mismatches with the extracted truth, logs each replacement, and fails hard on a corrupted entry instead of repairing it. A reviewer finding's `current` block was adjudicated by consensus, the red team, and the report without anyone checking the quoted code exists. `verify-finding-evidence.sh` runs over each persisted shard result before the merge. A kept finding whose non-empty `current` does not occur in the target file under whitespace normalization moves to `contested` with an outcome naming the failed match, and the demotion is synced into `adversarial_input`, so a fabricated quote never reaches the merge, the red team, or the report as kept. A finding that quoted no code is exempt. The rendered report now labels every finding with the scrutiny it actually received, `adversary-tested` or `consensus-only`, derived from which stage produced the finding's final state. The new `fix-application.md` reference carries the contract for whoever applies a report's remediations: apply `suggested` verbatim, self-review the fix diff for redundancy and tautological tests plus the static gates, judge mutants against the whole suite only, re-verify the premise of a `consensus-only` must-fix finding before applying it, and confirm reported commits with `git merge-base --is-ancestor` before calling them landed. Both gate scripts validate their input shape per entry and are covered by 18 BATS tests in plugin-tests/test-writing, wired into the skill's Phase 1 and Phase 5 with matching digraph nodes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Patch release carrying the adversarial-stage degradation fixes and the deterministic manifest and evidence gates; CHANGELOG entry derived against main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agent Skills ExportBuilt 7 skill packages: |
… tests Three fixtures quote PHP code containing `$x` and `$doesNotExist` in single quotes on purpose, and CI's shellcheck run over `.bats` files fails on the resulting SC2016 notes. Each site gets a per-line disable directive with the reason, matching the convention in `plugin-tests/mcp-shared/environment.bats`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
phpunit-test-team-reviewingpipeline survives a malformed defense stance instead of discarding the run, and stops trusting two agent-reported values it can check itself.test-writinggoes 5.1.0 to 5.1.1. Closes #173; implements the manifest-count, finding-evidence, fix-application, commit-reachability, scrutiny-label, and re-characterization items of #176 — the structure-only layout consolidation item is not part of this change.Defense stage degrades instead of aborting
A
mode=adversarialrun threw at the wave boundary when a defense reconciler listed an adversary-introduced finding under bothadopted_newandfindings: the maintained-findings fold resolved the id against the file's kept and contested records only, found nothing, and the throw discarded every completed agent in the run. The duplicated entry is now skipped as an already-cast vote (seenA/seenRare per-defender, so they answer exactly whether this defender already voted on the id), anddefensePromptstates that the four response arrays are disjoint, removing the ambiguity that produced the duplication in the first place.Beyond that fast path, any defense stance entry failing an integrity guard — a malformed
finding_id, or an id resolving to no known record — takes the degrade-by-role path fromreferences/error-handling.mdthat already covered a dead defense reconciler: the offending entry is dropped, the rest of that stance still votes, the finding it named keeps its prior consensus binding, and each drop is recorded inred_team.defense_degraded(same null-when-clean shape ascoverage_gap) and rendered as a CAUTION block, so a degraded defense wave is never presented as a complete one. A dropped promotion is credited in no metric.Re-characterizations survive the merge
A defender can maintain a finding while correcting its location, method, and remediation after re-reading the file. The merge picked the descriptive owner by remediation length, so a terser correction lost every corrected field back to the stale original and its fix landed behind the original's in
suggested_variants— a fix applier working from the report would change code that was never defective. Maintained findings now merge throughrecharacterize(), which makes the defender's payload the descriptive owner whenever it proposed a remediation. A payload without one keeps the original's fields, preserving the invariant thatcurrentandsuggesteddescribe one change.Deterministic gates on agent-reported data
Two values flowed through the pipeline unvalidated although a script can check both. The gates run skill-side between stages, because the workflow script is filesystem-sandboxed and carrying file bodies in the manifest would breach the run-script size cap.
workflow/verify-method-counts.shmethod_count/test_methodsis replaced with the extracted truth and logged, never merely warnedworkflow/verify-finding-evidence.shcurrentblock does not occur in the target file under whitespace normalization is demoted tocontestedwith an outcome naming the failed match; findings that quoted no code are exemptThe evidence gate also syncs each demotion into the file's
adversarial_input, so a fabricated quote never reaches the merge, the red team, or the report as kept — one invocation covers Phase 5 and Phase 6. Both scripts validate their input shape per entry and fail hard on a corrupted manifest or result instead of repairing it.Scrutiny labels and the fix-application contract
The report never distinguished findings that survived a red team from findings nothing ever challenged. Every rendered finding now carries
Scrutiny: adversary-tested | consensus-only, derived from which stage produced the file's final per-finding state — never asserted per finding by any agent.The new
references/fix-application.mdis the contract for whoever applies a report's remediations: apply the report's finalsuggestedverbatim and keepsuggested_variantsas human-facing alternatives; self-review the fix diff before committing (DESIGN-003/DESIGN-004 across co-touched files, a tautology check on every added test, and the same PHPStan/php-cs-fixer gates the reviewed code passes through); judge any mutant against the whole suite, never a single test; re-verify the factual premise of aconsensus-onlymust-fix finding before applying it; and verify every reported commit withgit merge-base --is-ancestorbefore calling it landed.