Fail closed on unattributed batch failures - #63
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
unittestregression where an imported failingTestCaseproduces an unattributed failure while another target fails exactly and a third target is unavailableWhy
parseTargetObservations()previously downgraded otherwise-passing observations forunattributedFailures > 0only when no qualified target was unavailable. That made the evidence boundary non-monotonic: adding an unrelated unavailable target could make an ambiguous passing target look exact again.A normal
unittestfixture reproduces this without relying on adversarial control-pipe behavior. A target can import aTestCasedefined in another module; unittest executes it, but the observer cannot attribute that failure to the target path, sounattributedFailuresincreases. If a separate target is unavailable, the old parser allowed other nonfailed observations to remainpassed. Those observations can then participate in file-level verification even though the process-level failure cannot be excluded from them.The fail-closed rule is now simple: once the runner reports an unattributed process-level failure, every nonfailed target observation becomes
not-observed; only explicitly failed targets keep localized failure attribution.Verification
unittestpayload with one observed pass, one exact failure, one unavailable target, and one unattributed failurenpm testpassed after the changenpm run clean && npm run buildpassedgit diff --checkpassed