Skip to content

fix(proof): a failing proof must fail the job; classify terse denials; probe an existing bucket - #20

Merged
DustyStudy merged 1 commit into
mainfrom
fix/prove-honesty
Sep 21, 2026
Merged

DustyStudy merged 1 commit into
mainfrom
fix/prove-honesty

Conversation

@DustyStudy

Copy link
Copy Markdown
Owner

What the first real run of prove.yml showed

All 7 self-escalation calls hit the explicit deny exactly as designed, and all 4 controls worked. But the run also exposed three defects in my proof - one of them serious.

1. A failing proof went green (the serious one)

The output said RESULT: FAIL and the job succeeded. GitHub's default shell for run: is bash -e, which has no pipefail, so python ... | tee reported tee's success and swallowed the script's exit status. My own test even asserted "pipefail is on" in a comment - it wasn't. A proof that cannot fail proves nothing.

Fix: shell: bash (which runs bash -eo pipefail) on that step, and a test that requires it.

$ bash -e -c 'false | tee /dev/null'            -> exit 0   (failure swallowed)
$ bash -eo pipefail -c 'false | tee /dev/null'  -> exit 1   (failure survives the pipe)

2. delete_ou / move_account read as "denied for some other reason" - but the role was right

CloudTrail's record of those calls: "...not authorized to perform: organizations:MoveAccount ... because no identity-based policy allows the organizations:MoveAccount action" - an implicit deny, as designed. The Organizations API just tells the caller less than IAM logs, which my classifier didn't expect.

Fix: a denied_generic verdict. outside_scope accepts implicit or generic; self_escalation still demands a confirmed explicit deny (a terse denial can't be told from an explicit one, so it never satisfies that check). SCP denials stay a separate, non-crediting verdict.

3. read_another_bucket read as "authorized" - a flawed probe

S3 answers NoSuchBucket for a nonexistent bucket before it checks permissions, so such a probe looks authorized whatever the role may do. Replaced with change_state_bucket_policy: the existing state bucket, with a malformed policy ({}) S3 rejects if the call were ever wrongly allowed. Checked against the IAM simulator first (implicitDeny). A test now forbids S3 probes against buckets that don't exist.

Verification

  • 5 new/changed tests (8 failed before the fix); 154 pass. ruff, bandit, actionlint clean.
  • Root causes established from CloudTrail's own error text and the simulator, not guessed.
  • The bash -e vs bash -eo pipefail behaviour demonstrated above.

Not verified yet

The re-run. After this merges, prove.yml should now (a) fail the job if any probe disagrees, and (b) pass with 17 verdicts. If it fails, that's a real finding.

…; probe an existing bucket

The first real run of prove.yml printed RESULT: FAIL and went green. GitHub's default shell for run: is bash -e, which has no pipefail, so python | tee reported tee's success. The step now sets shell: bash (pipefail), and a test pins it - a proof that cannot fail proves nothing.

The three mismatches were the probes, not the role. CloudTrail records IAM's own answer for the two Organizations calls as 'because no identity-based policy allows ...' (implicit deny, as designed), but the Organizations API tells the caller only that it lacks permission, which the classifier read as 'denied for some other reason'. Added denied_generic: outside-scope accepts implicit or generic, while self-escalation still demands a confirmed explicit deny. And S3 answers NoSuchBucket for a nonexistent bucket before checking permissions, so the bucket probe read as authorized; it now targets the existing state bucket with a malformed policy that S3 rejects if the call were ever wrongly allowed. Tests pin all three.
@DustyStudy
DustyStudy merged commit b8d50b9 into main Sep 21, 2026
5 checks passed
@DustyStudy
DustyStudy deleted the fix/prove-honesty branch September 21, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant