fix(proof): a failing proof must fail the job; classify terse denials; probe an existing bucket - #20
Merged
Merged
Conversation
…; probe an existing bucket The first real run of prove.yml printed RESULT: FAIL and went green. GitHub's default shell for run: is bash -e, which has no pipefail, so python | tee reported tee's success. The step now sets shell: bash (pipefail), and a test pins it - a proof that cannot fail proves nothing. The three mismatches were the probes, not the role. CloudTrail records IAM's own answer for the two Organizations calls as 'because no identity-based policy allows ...' (implicit deny, as designed), but the Organizations API tells the caller only that it lacks permission, which the classifier read as 'denied for some other reason'. Added denied_generic: outside-scope accepts implicit or generic, while self-escalation still demands a confirmed explicit deny. And S3 answers NoSuchBucket for a nonexistent bucket before checking permissions, so the bucket probe read as authorized; it now targets the existing state bucket with a malformed policy that S3 rejects if the call were ever wrongly allowed. Tests pin all three.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the first real run of
prove.ymlshowedAll 7 self-escalation calls hit the explicit deny exactly as designed, and all 4 controls worked. But the run also exposed three defects in my proof - one of them serious.
1. A failing proof went green (the serious one)
The output said
RESULT: FAILand the job succeeded. GitHub's default shell forrun:isbash -e, which has nopipefail, sopython ... | teereportedtee's success and swallowed the script's exit status. My own test even asserted "pipefail is on" in a comment - it wasn't. A proof that cannot fail proves nothing.Fix:
shell: bash(which runsbash -eo pipefail) on that step, and a test that requires it.2.
delete_ou/move_accountread as "denied for some other reason" - but the role was rightCloudTrail's record of those calls: "...not authorized to perform: organizations:MoveAccount ... because no identity-based policy allows the organizations:MoveAccount action" - an implicit deny, as designed. The Organizations API just tells the caller less than IAM logs, which my classifier didn't expect.
Fix: a
denied_genericverdict.outside_scopeaccepts implicit or generic;self_escalationstill demands a confirmed explicit deny (a terse denial can't be told from an explicit one, so it never satisfies that check). SCP denials stay a separate, non-crediting verdict.3.
read_another_bucketread as "authorized" - a flawed probeS3 answers
NoSuchBucketfor a nonexistent bucket before it checks permissions, so such a probe looks authorized whatever the role may do. Replaced withchange_state_bucket_policy: the existing state bucket, with a malformed policy ({}) S3 rejects if the call were ever wrongly allowed. Checked against the IAM simulator first (implicitDeny). A test now forbids S3 probes against buckets that don't exist.Verification
bash -evsbash -eo pipefailbehaviour demonstrated above.Not verified yet
The re-run. After this merges,
prove.ymlshould now (a) fail the job if any probe disagrees, and (b) pass with 17 verdicts. If it fails, that's a real finding.