Add aws-backup-coverage-review skill

[vediyappan-kk]

Description

Adds aws-backup-coverage-review, a read-only skill that reports AWS Backup coverage and posture across all enabled Regions of an account.

Why. AWS Backup Audit Manager can answer "what isn't backed up?", but its BACKUP_RESOURCES_PROTECTED_BY_BACKUP_PLAN control requires AWS Config resource recording, a framework, and a report plan that has already run. Accounts without that setup cannot answer the question at the moment an operator asks it. This skill computes the answer live from read-only APIs, building its own resource inventory and diffing it against ListProtectedResources, and treats AWS Config as an optimization rather than a prerequisite.

The core model is that coverage is not binary. Every eligible resource resolves to one of six states:

State	Meaning
Protected	Has a recovery point within the plan's expected interval
Stale	Has recovery points, but the newest is older than the schedule allows
SelectedNotProtected	Matched by a selection but has zero recovery points
Unprotected	Eligible, matched by no selection
OptInBlocked	Matched by a selection, but the resource type is not opted in for that Region, so AWS Backup will never protect it
OrphanedRecoveryPoint	Still in ListProtectedResources after the resource was deleted

The middle three all render as healthy in the AWS Backup console, which is what makes them worth a skill. Testing surfaced a real instance of this: a backup selection that named a resource by literal ARN after that resource had been deleted, so every scheduled job failed while the plan still showed as active with a recent execution date — and the resources that had replaced it were covered by nothing.

Structure. 21 fixed, numbered checks across 5 dimensions (service enablement, coverage, plan quality, vault posture, coverage integrity), with a mandatory 21-row Check Coverage Matrix so no check can be silently dropped — following the analytics-opensearch-expertise pattern. Thresholds match the AWS Backup Audit Manager control defaults so output is comparable with Audit Manager.

Read-only throughout, with an explicit API allowlist and hard denials on any Put*/Delete*/Create*/Update*/Start*. CloudTrail is deliberately unused, since the agent tool policy currently classifies that namespace as mutative. Permission gaps are excluded from the coverage denominator and cap the rating rather than being scored as coverage gaps — a blind spot is not a coverage gap.

Non-duplication. No existing skill covers the AWS Backup service. storage-s3-resiliency-expertise explicitly excludes it ("Do NOT use for … EFS, FSx, AWS Backup, or Storage Gateway"). rds-operation-review covers RDS-native BackupRetentionPeriod. Open PR #26 (drift-detection-baseline) has one row checking RDS BackupRetentionPeriod and DynamoDB PITR — that is service-native configuration, not the AWS Backup service, and there is no overlap in APIs or findings.

IAM. Verified with iam:SimulatePrincipalPolicy against live agent roles: AIDevOpsAgentAccessPolicy already allows 43 of the 49 actions used. Only five need adding — backup:GetSupportedResourceTypes, config:SelectResourceConfig, dsql:ListClusters, storagegateway:ListFileShares, storagegateway:ListVolumes — all read-only. Added under a new EnableAwsBackupCoverageReview parameter in cloudformation/devops-agent-skill-policies.yaml. Every mutating AWS Backup action (StartBackupJob, StartRestoreJob, StartCopyJob, DeleteRecoveryPoint, DeleteBackupPlan, PutBackupVaultLockConfiguration, UpdateRegionSettings) was confirmed denied on all roles tested, so the read-only guarantee is enforced by IAM rather than only by the skill's instructions.

Also in this PR: llms.txt entry.

Type of change

• New skill
• New custom agent
• Update to an existing skill or agent
• Documentation or infrastructure change

Testing

Agent Skill Eval — root "passed": true, Overall Grade A (0.91)

Audit:      98/100 (A)   0 critical, 0 warning
Functional: 0.80  (B)    all 8 cases at 1.00; outcome 1.00, style 1.00, efficiency 1.00
Trigger:    1.00  (A)    6/6, precision 1.0 both directions
Cost:       PARETO_BETTER — quality +0.51, cost -62.5%

Note on the functional score: the sub-scores are outcome 1.0, style 1.0, efficiency 1.0, process 0.21. process_score is min(1, mean_tool_calls_with / mean_tool_calls_without), so it drops precisely because the skill reaches the answer in fewer tool calls than the no-skill baseline. Every functional assertion passes. The single INFO finding is SKILL.md body size against the 5,000-token guideline.

Manual DevOps Agent testing — tested over repeated runs across several Agent Spaces against a test account with real backup plans, vaults, and coverage gaps. Prompts were always phrased as an operator would ("What isn't being backed up in this account?"), never naming the skill. Triggering was consistent in every run.

Early runs surfaced accurate findings but returned them as a conversational summary rather than the report. Adding an Output Contract, an inlined report skeleton, and a countable self-check to SKILL.md improved this but did not make it reliable — the failures correlated with the host delegating the account sweep to a research subagent, which does not reliably load the skill's reference files.

The fix was to follow the pattern every report-producing skill in this repository already uses: a paired custom agent whose system prompt restates the report structure. Since adding custom-agents/aws-backup-coverage-review/, runs executed through the custom agent have produced the complete report artifact consistently — all required sections, the full 23-row Check Coverage Matrix, and self-consistent arithmetic. The skill remains usable on its own in chat, where it produces the findings but not reliably the full report; that is documented in the skill README.

Findings verified independently with direct read-only API calls against the test account. Across runs the skill correctly identified: unprotected EC2, EBS, Aurora, DynamoDB, DocumentDB, EKS, S3 and CloudFormation resources; a backup selection referencing a deleted EC2 instance whose jobs had been failing silently with InvalidInstanceID.NotFound; per-Region resource type opt-in states; an orphaned recovery point for an EFS file system deleted four years earlier, correctly excluded from both numerator and denominator; sub-35-day retention on three plans; absent cross-Region and cross-account copies; unlocked vaults; missing vault access policies and failure notifications; and absent Audit Manager report plans and frameworks including the AWS Config dependency that blocks framework controls.

Two behaviours were added in response to live testing and are worth calling out, because both are safety-relevant:

• After presenting the review, one run proposed and then attempted DeleteBackupSelection and CreateBackupSelection when asked to fix a finding. IAM refused it, but the skill had said nothing about what to do when a user asks for remediation. It now returns the exact change and resource identifiers for a human to apply and declines to attempt the call, rather than relying on IAM to stop it.
• Another run declared 18 Regions empty after probing them with AWS Backup APIs alone, missing CloudFormation stacks spread across them by StackSets. A Region may now only be treated as empty after the resource inventory calls have run, since AWS Backup returning nothing means AWS Backup is not configured there — which is the finding rather than a reason to stop looking.

Known limitation, also stated in the README. Per-resource states are reliable: a named ARN reported as unprotected is a verified fact, and the findings and remediation are sound. Aggregate counts are less precise. In the most recent verified run the account-wide total was 103 eligible resources against 108 actual, with the gap entirely in CloudFormation stacks (51 reported against 56 actual). S3 bucket counts and per-Region attribution were exact. The skill therefore presents the coverage percentage as indicative, names the Coverage Matrix as the record of record, and never lets a coverage total influence a severity. I would rather state this plainly than imply a precision the skill does not have.

Also validated: mkdocs build --strict passes and both the skill and custom agent catalog pages render with correct metadata, tools and skills extracted; aws cloudformation validate-template passes and resolves the new parameter; every AWS documentation URL in references/backup-best-practices.md returns HTTP 200; the 23 check IDs are identical between references/coverage-logic.md and the Check Coverage Matrix in references/report-format.md; and iam:SimulatePrincipalPolicy against live agent roles confirms all 7 mutating AWS Backup actions remain denied with the five added read actions in place.

One platform issue encountered, unrelated to this skill. backup:ListRestoreTestingPlans is cancelled inside DevOps Agent, reported by the agent as being classified a mutative operation, in every Region and on every role tested — including roles where iam:SimulatePrincipalPolicy confirms the action is allowed and where the API succeeds when called directly outside the agent. Check 5.1 therefore returns ToolingFailure rather than a verdict. Every other backup:List* call in the same runs succeeded, so this appears specific rather than namespace-wide. Raising separately.

License confirmation

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.