Skip to content

Fail closed on unattributed batch failures - #63

Merged
hzw0813 merged 8 commits into
mainfrom
fix/unattributed-target-failure-sink
Aug 15, 2026
Merged

Fail closed on unattributed batch failures#63
hzw0813 merged 8 commits into
mainfrom
fix/unattributed-target-failure-sink

Conversation

@hzw0813

@hzw0813 hzw0813 commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • treat any unattributed targeted-runner process failure as ambiguous for every non-failed exact target observation in that batch
  • prevent an unrelated unavailable target from acting as a sink that lets another observed pass survive a process-level failure
  • preserve explicitly localized exact target failures as negative evidence
  • add a real stdlib unittest regression where an imported failing TestCase produces an unattributed failure while another target fails exactly and a third target is unavailable
  • strengthen the existing partially-localized pytest regression so an independently observed pass is withheld when the process also contains an unattributed failure
  • regenerate tracked dist and document the change

Why

parseTargetObservations() previously downgraded otherwise-passing observations for unattributedFailures > 0 only when no qualified target was unavailable. That made the evidence boundary non-monotonic: adding an unrelated unavailable target could make an ambiguous passing target look exact again.

A normal unittest fixture reproduces this without relying on adversarial control-pipe behavior. A target can import a TestCase defined in another module; unittest executes it, but the observer cannot attribute that failure to the target path, so unattributedFailures increases. If a separate target is unavailable, the old parser allowed other nonfailed observations to remain passed. Those observations can then participate in file-level verification even though the process-level failure cannot be excluded from them.

The fail-closed rule is now simple: once the runner reports an unattributed process-level failure, every nonfailed target observation becomes not-observed; only explicitly failed targets keep localized failure attribution.

Verification

  • reproduced the stdlib unittest payload with one observed pass, one exact failure, one unavailable target, and one unattributed failure
  • full npm test passed after the change
  • clean npm run clean && npm run build passed
  • git diff --check passed
  • tracked dist regenerated
  • temporary updater workflows removed from the final diff

@hzw0813
hzw0813 merged commit deb3256 into main Aug 15, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant