Skip to content

feat(proof): script that shows the guardrail SCP enforces in a real member account - #18

Merged
DustyStudy merged 2 commits into
mainfrom
feat/prove-enforcement
Sep 21, 2026
Merged

DustyStudy merged 2 commits into
mainfrom
feat/prove-enforcement

Conversation

@DustyStudy

Copy link
Copy Markdown
Owner

What

cli/prove_enforcement.py: a script that shows the guardrail SCP actually enforces in a real member account, by making the same calls in each phase so the only variable is the SCP - denied while it's attached, allowed before and after. That before/after toggle is what shows the SCP (and not a missing permission or a typo) was the cause.

It creates nothing: EC2 launches are dry runs, and the CloudTrail / Config calls target a trail and a recorder that don't exist, so an authorized call fails with "not found" and a denied one with AccessDenied.

Design points

  • Only an AccessDenied that names a service control policy counts as enforcement. A denial for any other reason is classified denied_other and fails the phase - it would otherwise look exactly like the SCP working.
  • Controls. A call that must always work (imdsv2_launch, and a launch on an IMDSv2-by-default image) guards against a broken account invalidating the rest.
  • Implicit launches are asserted both ways. The SCP resolves ec2:MetadataHttpTokens from the image when the request leaves it unset: an IMDSv2-by-default image (AL2023) is allowed (the instance is IMDSv2), an image that defaults to IMDSv1 is denied. I was unsure this case was covered, so it is now tested rather than assumed.
  • The legacy-image probe is skipped, not failed, if that public AMI parameter is ever retired.
  • --wait retries while an SCP change propagates. Session is injected so the logic is testable; exit code is non-zero on mismatch.

Also: fix(workflow) - the apply summary now knows a destroy has no outputs (it labelled a destroy "Applied" and left a confusing output not found error, masked by || true).

Evidence it produces (from the first real run, against a throwaway OU with one test account)

phase guardrail calls (4) controls (2)
SCP applied through the terraform workflow all denied by an SCP allowed
after destroy through the same workflow all allowed allowed

The apply and destroy each went through the two-approval workflow as orgseed-ci; the org afterwards had only FullAWSAccess, no baseline SCP, and the test account still in its OU.

Verification

  • 19 tests for the classification and the phase logic (written first; they failed before the script existed); 113 total. ruff, bandit, actionlint clean.
  • The script itself was run for real in both phases (above).

Not covered

This proves enforcement in one account of one org. It does not prove GovCloud, a second org, or anything about accounts other than the test account.

…ember account

prove_enforcement.py makes the same handful of calls in a test account in each phase (no SCP / SCP attached), so the only variable is the SCP: denied while it is attached, allowed before and after. Only an AccessDenied that names a service control policy counts as enforcement; a denial for any other reason is reported as such and fails the phase. A control call that must always work guards against a broken account. Nothing is created (EC2 launches are dry runs; CloudTrail and Config calls target things that do not exist).

A launch that sets no metadata options is asserted both ways: an IMDSv2-by-default image (AL2023) is allowed, an image that defaults to IMDSv1 is denied - the SCP resolves the metadata setting from the image, so it catches the implicit case too. The legacy-image probe is skipped, not failed, if that public AMI parameter is ever retired.
After a destroy there are no outputs, so the summary read them anyway (masked by || true) and labelled the run Applied, leaving a confusing 'output not found' error in the log. It now says Destroyed, and the error-swallowing is gone so a genuine failure to read outputs on an apply is visible.
@DustyStudy
DustyStudy merged commit e399fc0 into main Sep 21, 2026
5 checks passed
@DustyStudy
DustyStudy deleted the feat/prove-enforcement branch September 21, 2026 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant