From d018b4dcb200bab6b0654a737641b0ebfa0f9d03 Mon Sep 17 00:00:00 2001 From: Sean Nixon Date: Fri, 4 Sep 2026 09:39:08 -0700 Subject: [PATCH] Add bedrock-operation-review skill and include in aws-operation-review agent --- .../aws-operation-review/SYSTEM_PROMPT.md | 5 +- .../bedrock-operation-review/.skilleval.yaml | 8 + skills/bedrock-operation-review/CHANGELOG.md | 16 + skills/bedrock-operation-review/README.md | 147 ++++ skills/bedrock-operation-review/SKILL.md | 358 ++++++++++ .../evals/benchmark.json | 629 ++++++++++++++++++ .../evals/eval_queries.json | 8 + .../bedrock-operation-review/evals/evals.json | 62 ++ .../evals/files/bedrock-context.json | 6 + .../evals/report.json | 65 ++ .../evals/trigger_report.json | 94 +++ .../references/best-practices-checklist.md | 55 ++ .../references/metrics-thresholds.md | 83 +++ 13 files changed, 1534 insertions(+), 2 deletions(-) create mode 100644 skills/bedrock-operation-review/.skilleval.yaml create mode 100644 skills/bedrock-operation-review/CHANGELOG.md create mode 100644 skills/bedrock-operation-review/README.md create mode 100644 skills/bedrock-operation-review/SKILL.md create mode 100644 skills/bedrock-operation-review/evals/benchmark.json create mode 100644 skills/bedrock-operation-review/evals/eval_queries.json create mode 100644 skills/bedrock-operation-review/evals/evals.json create mode 100644 skills/bedrock-operation-review/evals/files/bedrock-context.json create mode 100644 skills/bedrock-operation-review/evals/report.json create mode 100644 skills/bedrock-operation-review/evals/trigger_report.json create mode 100644 skills/bedrock-operation-review/references/best-practices-checklist.md create mode 100644 skills/bedrock-operation-review/references/metrics-thresholds.md diff --git a/custom-agents/aws-operation-review/SYSTEM_PROMPT.md b/custom-agents/aws-operation-review/SYSTEM_PROMPT.md index 57808d5..a24fe04 100644 --- a/custom-agents/aws-operation-review/SYSTEM_PROMPT.md +++ b/custom-agents/aws-operation-review/SYSTEM_PROMPT.md @@ -2,7 +2,7 @@ You are an AWS Operations Review Specialist focused on assessing AWS services ag ## Goal -Perform comprehensive operational reviews of AWS services (EKS clusters, RDS instances, Aurora clusters) to identify gaps in security, reliability, performance, cost optimization, and operational excellence — aligned with AWS best practices and the Well-Architected Framework. +Perform comprehensive operational reviews of AWS services (EKS clusters, RDS instances, Aurora clusters, Bedrock workloads) to identify gaps in security, reliability, performance, cost optimization, and operational excellence — aligned with AWS best practices and the Well-Architected Framework. ## Approach @@ -10,6 +10,7 @@ Perform comprehensive operational reviews of AWS services (EKS clusters, RDS ins 2. Load the appropriate skill for the service: - For EKS clusters: use the `eks-operation-review` skill methodology - For RDS/Aurora databases: use the `rds-operation-review` skill methodology + - For Bedrock workloads: use the `bedrock-operation-review` skill methodology 3. Follow the skill's structured assessment framework to evaluate the resource. 4. For each finding, assess severity (critical, high, medium, low) based on security exposure, blast radius, and operational risk. 5. Generate actionable recommendations with clear remediation steps. @@ -40,7 +41,7 @@ Before creating new recommendations, list existing recommendations and update an Generate a shareable report artifact as a Markdown document. **Artifact naming:** `-review--.md` -Examples: `eks-review-prod-cluster-2026-06-21.md`, `rds-review-orders-db-2026-06-21.md` +Examples: `eks-review-prod-cluster-2026-06-21.md`, `rds-review-orders-db-2026-06-21.md`, `bedrock-review-1234567890-us-east-1-2026-08-21.md` **Report structure:** diff --git a/skills/bedrock-operation-review/.skilleval.yaml b/skills/bedrock-operation-review/.skilleval.yaml new file mode 100644 index 0000000..f20ad7a --- /dev/null +++ b/skills/bedrock-operation-review/.skilleval.yaml @@ -0,0 +1,8 @@ +audit: + ignore: + - STR-016 # README alongside SKILL.md is intentional + # Whitelist documentation/blog domains for best practices reference (won't trigger SEC-002) + safe_domains: + - aws.amazon.com + - docs.aws.amazon.com + - repost.aws \ No newline at end of file diff --git a/skills/bedrock-operation-review/CHANGELOG.md b/skills/bedrock-operation-review/CHANGELOG.md new file mode 100644 index 0000000..adc2284 --- /dev/null +++ b/skills/bedrock-operation-review/CHANGELOG.md @@ -0,0 +1,16 @@ +# Changelog + +## [1.0.0] - 2026-07-XX +### Added +- Initial release adapted from AWS Support Specialist skill +- Comprehensive Amazon Bedrock operational review aligned with the AWS Well-Architected Framework and Bedrock best practices +- Five review pillars: Security, Performance, Service Quotas, Cost Optimization, and Resilience +- Resource discovery across foundation models, guardrails, inference profiles, prompt routers, provisioned throughput, custom models, agents, knowledge bases, data sources, and Prompt Management +- CloudWatch metric collection and threshold-based classification (Normal/Warning/Critical) in the `AWS/Bedrock` namespace +- Service quota utilization analysis (RPM/TPM including CRIS) via Service Quotas API +- Cost optimization checks: prompt caching, model distillation, batch inference, provisioned throughput, intelligent prompt routing, and self-managed EC2 GPU utilization +- Cross-Region Inference (CRIS) adoption analysis for resilience +- Severity-ranked findings (CRITICAL, HIGH, MEDIUM, LOW, INFO) and a shareable Markdown report artifact +- AWS-API-only data collection (Bedrock, Bedrock Agent, CloudWatch, Service Quotas, EC2) with no data-plane model invocations and no prompt/response content read +- Reference files: best-practices checklist and CloudWatch metric thresholds +- Evaluation test cases (5 functional evals, 6 trigger queries) diff --git a/skills/bedrock-operation-review/README.md b/skills/bedrock-operation-review/README.md new file mode 100644 index 0000000..43a2198 --- /dev/null +++ b/skills/bedrock-operation-review/README.md @@ -0,0 +1,147 @@ +# Bedrock Operational Review — AWS DevOps Agent Skill + +A comprehensive Amazon Bedrock operational review skill for [AWS DevOps Agent](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent.html). Conducts best-practices assessments aligned with the [Amazon Bedrock User Guide](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) and the [AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html). Generates a shareable report artifact for the review. + +## What It Does + +When activated via Chat, this skill instructs the DevOps Agent to: + +1. Discover Bedrock resources in the configured account/regions — foundation models, guardrails, inference profiles, prompt routers, provisioned throughput, custom models, model customization jobs, agents, knowledge bases, data sources, and prompts. +2. Collect CloudWatch metrics from the `AWS/Bedrock` namespace (also `CWAgent` namespaces) for invocations, latency, throttling, errors, token usage, guardrail interventions, and prompt-cache activity. +3. Pull service quota values from Service Quotas and compare against observed usage. +4. Analyze against six pillars — **Security, Performance, Service Quotas, Cost Optimization, Resilience** — plus check-specific guidance. +5. Generate a shareable report artifact, named `bedrock-review---.md`. + +All data is gathered through native AWS APIs (`bedrock`, `bedrockagent`, `cloudwatch`, `servicequotas`, `ec2`). The skill performs **no data-plane model invocations** and reads no prompt or response content. It does not depend on Kubernetes, EKS, or any internal tooling. + +## Agent Types + +This skill is intended for the following agent types (selected in the Operator Web App at upload time): + +- **On-demand** — conversational invocation in Chat ("review my Bedrock account", "Bedrock health check"). +- **Evaluation** — proactive operational improvement recommendations. + +Select **Generic** instead if you want the skill available to all agent types. + +## Prerequisites + +### 1. An AWS DevOps Agent Space with the target AWS account + +You need an existing [Agent Space](https://docs.aws.amazon.com/devopsagent/latest/userguide/getting-started-with-aws-devops-agent-creating-an-agent-space.html) with the target AWS account configured as a cloud source. + +### 2. IAM permissions for the DevOps Agent's primary cloud-source role + +The Agent Space's IAM role must have read access to Bedrock, Bedrock Agent, CloudWatch, Service Quotas, and EC2 APIs. Verify these are present in your account before running the review: + +- `bedrock:ListFoundationModels`, `bedrock:ListGuardrails`, `bedrock:GetGuardrail` +- `bedrock:GetModelInvocationLoggingConfiguration` +- `bedrock:ListInferenceProfiles`, `bedrock:GetInferenceProfile` +- `bedrock:ListPromptRouters`, `bedrock:GetPromptRouter` +- `bedrock:ListProvisionedModelThroughputs`, `bedrock:GetProvisionedModelThroughput` +- `bedrock:ListCustomModels`, `bedrock:GetCustomModel` +- `bedrock:ListModelCustomizationJobs`, `bedrock:GetModelCustomizationJob` +- `bedrock:ListAgents`, `bedrock:GetAgent`, `bedrock:ListAgentAliases` (Bedrock Agent control plane) +- `bedrock:ListKnowledgeBases`, `bedrock:GetKnowledgeBase`, `bedrock:ListDataSources`, `bedrock:GetDataSource` +- `bedrock:ListPrompts`, `bedrock:GetPrompt` +- `cloudwatch:ListMetrics`, `cloudwatch:GetMetricData`, `cloudwatch:GetMetricStatistics` +- `servicequotas:GetServiceQuota`, `servicequotas:ListServiceQuotas` +- `ec2:DescribeInstances` + +The skill operates entirely in **read-only** mode: it never calls `Create*`, `Update*`, `Delete*`, or any `InvokeModel*` (data-plane) APIs. + +### 3. Model invocation activity (recommended) + +Most CloudWatch-based checks (latency, throttling, prompt caching, guardrail signals, cross-region inference, model versions) rely on `AWS/Bedrock` metrics, which only exist for models that have been invoked in the analysis window. Reviewing an account with no recent Bedrock traffic still produces a configuration report, but metric-driven findings will be empty. + +### 4. (Conditional) CloudWatch Agent for EC2 GPU utilization + +The EC2 GPU utilization check (self-managed P4/P5/P5en/P6 instances) requires the **CloudWatch Agent** installed with the **NVIDIA DCGM plugin** enabled, publishing `nvidia_smi_utilization_gpu` and `nvidia_smi_memory_util` to the `CWAgent` namespace. Without it, GPU signals cannot be evaluated. This check is optional and only applies to accounts running self-managed GPU training/inference. + +## Uploading to AWS DevOps Agent + +> Reference: [Uploading a skill](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#uploading-a-skill) + +### 1. Package the skill + +From the `skills/` directory in this repo: + +```bash +cd skills +zip -r bedrock-operation-review.zip bedrock-operation-review/ -x 'bedrock-operation-review/evals/*' +``` + +The resulting `bedrock-operation-review.zip` contains: + +``` +bedrock-operation-review/ +├── SKILL.md # frontmatter + skill instructions (required) +├── README.md +└── references/ + ├── best-practices-checklist.md + └── metrics-thresholds.md +``` + +`evals/` is excluded from the upload to keep the zip small (it's only used for offline evaluation). + +Constraints (enforced at upload time): + +- Total zip size ≤ **6 MB**. +- `SKILL.md` is required and must include `name` and `description` frontmatter. +- A `scripts/` directory is **not** allowed — uploads containing scripts are rejected. + +### 2. Upload via the Operator Web App + +1. Navigate to the **Skills** page in your Agent Space Operator Web App. +2. Click **Add skill** → **Upload skill**. +3. Drag and drop `bedrock-operation-review.zip` (or browse to it). +4. Select agent types: **On-demand** and **Evaluation** (or leave **Generic** to make it available to all agent types). +5. Review the validation results. +6. Click **Upload**. + +## Usage + +In the DevOps Agent Chat, use natural language: + +- *"Run a Bedrock operational review for all regions."* +- *"Review my Bedrock account `123456789012` in `us-east-1` for best practices."* +- *"Audit Bedrock security and cost optimization."* +- *"Check my Bedrock service quota utilization and throttling."* +- *"ORR for our Bedrock workloads."* + +The agent will: + +- Collect all data automatically (no prompts for confirmation). +- Use only AWS APIs — no model invocations, no prompt/response content read. +- Generate a report artifact named `bedrock-review---.md`. + +## Skill Contents + +``` +bedrock-operation-review/ +├── SKILL.md # main skill instructions (with frontmatter) +├── README.md # this file +├── references/ +│ ├── best-practices-checklist.md # checklist mapped to Bedrock best practices +│ └── metrics-thresholds.md # CloudWatch metric thresholds & severity rules +└── evals/ # evaluation data (not included in upload zip) +``` + +## Best-Practices Pillars Covered + +| # | Pillar | Checks | Reference | +|---|--------|--------|-----------| +| 1 | Security | Guardrails, Guardrail Signals, Model Invocation Logging, Knowledge Base config & encryption, VPC config for customization jobs, IAM fine-grained access, KB logging, Prompt injection, Model access | [Bedrock security](https://docs.aws.amazon.com/bedrock/latest/userguide/security.html) | +| 2 | Performance | Latency & Throttling, Agent Performance, Invoked Model Versions, Data Automation, Service Tier | [Monitoring Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring.html) | +| 3 | Service Quotas | Model Quotas, Guardrail Service Quotas | [Bedrock quotas](https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html) | +| 4 | Cost Optimization | Application Inference Profiles, Custom Model Distillation, Prompt Caching, Prompt Management, Intelligent Prompt Routing, Provisioned Throughput, Batch Inference, EC2 GPU Utilization | [Bedrock pricing](https://aws.amazon.com/bedrock/pricing/) | +| 5 | Resilience | Cross-Region Inference (CRIS) | [Cross-region inference](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html) | + +## Severity Definitions + +| Severity | Definition | SLA | +|----------|------------|-----| +| CRITICAL | Immediate risk to availability, security, or data integrity | 24–48 hours | +| HIGH | Significant gap that could lead to incidents | 1 week | +| MEDIUM | Notable improvement opportunity | 30 days | +| LOW | Minor optimization or hardening | When convenient | +| INFO | Observation, no action required | N/A | diff --git a/skills/bedrock-operation-review/SKILL.md b/skills/bedrock-operation-review/SKILL.md new file mode 100644 index 0000000..e97bfc3 --- /dev/null +++ b/skills/bedrock-operation-review/SKILL.md @@ -0,0 +1,358 @@ +--- +name: bedrock-operation-review +description: Comprehensive Amazon Bedrock review aligned with the AWS + Well-Architected Framework and Bedrock best practices. Use this skill when + a user asks to review, audit, or assess Amazon Bedrock workloads for + best-practices compliance, security posture, performance, cost optimization, + service quotas, or resilience. Triggers on requests + like "Bedrock review", "Bedrock best practices audit", "GenAI operational + assessment", "review my Bedrock account", "Bedrock health check", "Bedrock cost + optimization review", or "ORR for Bedrock". +metadata: + author: smnixon + version: "1.0.0" + aws-devops-agent-skills.agent-types: "Chat tasks, Evaluation" + aws-devops-agent-skills.aws-services: "Amazon Bedrock" + aws-devops-agent-skills.technical-domains: "Machine Learning, GenAI" +--- + +# Amazon Bedrock Operational Review + +Conduct a comprehensive operational review of Amazon Bedrock workloads aligned with the +[AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html) +and [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) +best practices. + +This skill uses the **AWS Bedrock, Bedrock Agent, CloudWatch, Service Quotas, and +EC2 APIs only** — all data is collected through native AWS control-plane APIs and +CloudWatch metrics. It performs no data-plane model invocations and reads no prompt +or response content. + +## When to Use + +Activate this skill when the user asks to: +- Review, audit, or assess an Amazon Bedrock workload +- Check Bedrock best-practices compliance +- Evaluate Bedrock security, performance, cost, quotas, or resilience +- Perform a Bedrock operational readiness review (ORR) +- Investigate Bedrock configuration drift, throttling, or cost drivers + +## Step 1: Identify Target Scope + +Ask the user which accounts and regions to review. Accept: +- Specific account IDs and regions +- "all regions" for a given account +- A specific pillar or set of checks (e.g. "just cost optimization") + +If no scope is given, default to all configured account regions and all pillars. +Default the analysis window to the last 7 days unless the user specifies a range +(historical windows older than ~2 weeks may have reduced CloudWatch resolution). + +## Step 2: Discover Bedrock Resources + +Per account/region, begin by checking whether any Bedrock resources exist. Call +`bedrock.ListFoundationModels` and `bedrockagent.ListAgents` as a lightweight probe. +**If both return empty results and CloudWatch shows no `AWS/Bedrock` metrics for the +region, record "No Bedrock activity detected — skipping pillar analysis" for that +region and move on. Do not generate empty findings tables for inactive regions.** + +For regions where Bedrock is active, collect the full resource inventory: + +``` +bedrock.ListFoundationModels # available models + lifecycle status +bedrock.ListGuardrails / GetGuardrail # configured guardrails +bedrock.GetModelInvocationLoggingConfiguration +bedrock.ListInferenceProfiles / GetInferenceProfile +bedrock.ListPromptRouters / GetPromptRouter +bedrock.ListProvisionedModelThroughputs / GetProvisionedModelThroughput +bedrock.ListCustomModels / GetCustomModel +bedrock.ListModelCustomizationJobs / GetModelCustomizationJob +bedrockagent.ListAgents / GetAgent # agents (draft version) +bedrockagent.ListAgentVersions / GetAgentVersion # deployed versions — captures + # per-version model ID, orchestration + # type, and prompt override config +bedrockagent.ListAgentAliases +bedrockagent.ListKnowledgeBases / GetKnowledgeBase +bedrockagent.ListDataSources / GetDataSource +bedrockagent.ListPrompts / GetPrompt # Prompt Management +``` + +Capture per resource: identifiers, ARNs, status, creation/update timestamps, +encryption configuration (AWS-managed vs customer-managed KMS key), and any +VPC configuration. + +## Step 3: Collect CloudWatch Metrics (namespace `AWS/Bedrock`) + +**Before querying metrics, load the authoritative thresholds reference:** +``` +read_skill_resource(skill_id="bedrock-operation-review", path="references/metrics-thresholds.md") +``` +Use the thresholds from that file when classifying metric values as Normal, Warning, +or Critical throughout this step and Step 4. + +Discover which models are actually invoked with `cloudwatch.ListMetrics` +(dimension `ModelId`), then pull metric data with `cloudwatch.GetMetricData`. +Use the user's selected window; default to 7 days. + +Key model-level metrics (dimension `ModelId`): + +| Metric | Stat | Signal | +|--------|------|--------| +| Invocations | Sum | Volume / denominator for rate calcs | +| InvocationThrottles | Sum | Throttling / quota pressure | +| InvocationServerErrors | Sum | Server-side failures | +| InvocationClientErrors | Sum | Bad input / blocked content | +| InvocationLatency | Average, p95 | End-to-end latency | +| TimeToFirstToken | Average, p95 | Perceived latency (streaming) | +| InputTokenCount | Sum | Input token volume | +| OutputTokenCount | Sum | Output token volume | +| InvocationsIntervened | Sum | Guardrail interventions | +| TextUnitCount | Sum | Guardrail text-unit consumption | +| CacheReadInputTokenCount | Sum | Prompt cache reads | +| CacheWriteInputTokenCount | Sum | Prompt cache writes | + +GPU metrics for self-managed workloads live in the `CWAgent` namespace +(`nvidia_smi_utilization_gpu`, `nvidia_smi_memory_util`). + +`cloudwatch.GetMetricData` allows up to 500 metric-data queries per call — batch +requests and paginate when a region has many invoked models. + +## Step 4: Analyze Against Best Practices + +**Before evaluating findings, load the best-practices checklist:** +``` +read_skill_resource(skill_id="bedrock-operation-review", path="references/best-practices-checklist.md") +``` +Use the checklist as the canonical list of items to evaluate for each pillar. Mark +each item as ✅ Pass, ⚠️ Warning, ❌ Fail, or ➖ Not Applicable, and generate a +finding for every Fail or Warning. + +Evaluate all collected data across the pillars below and assign a severity to every +finding: CRITICAL, HIGH, MEDIUM, LOW, or INFO. The pillars mirror the Bedrock +operational review structure: **Security, Performance, Service Quotas, +Cost Optimization, Resilience.** + +### 4.1 Security +Ref: [Security in Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/security.html) + +- **Guardrails**: no guardrail configured on high-volume workloads → HIGH. Use + guardrails to filter hate, insult, sexual, violence, and PII content and to block + denied topics relevant to the use case. + [Guardrails](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html) +- **Guardrail signals**: high `InvocationClientErrors` (>5% of `Invocations`) + indicates problematic content reaching models — filter before invocation → MEDIUM. + `InvocationsIntervened` <5% of invocations on an active guardrail suggests + under-configuration; >15% suggests policy over-tuning → review. +- **Model invocation logging**: for workloads handling sensitive content (PII, PCI, + HIPAA), enabling invocation logging persists prompt/output on the account and may + conflict with data-handling requirements. Flag logging configuration mismatches → + MEDIUM. + [Model invocation logging](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html) +- **Knowledge Base configuration**: chunking (200–1000 tokens), embedding strategy, + encryption, access controls, and VPC endpoints → MEDIUM where missing. + [Knowledge bases](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html) +- **Knowledge Base data source encryption**: data sources without a customer-managed + KMS key when handling sensitive content → MEDIUM. + [Encryption of knowledge base resources](https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-kb.html) +- **VPC configuration for model customization jobs**: customization jobs without a + configured VPC expose training data to the internet → HIGH. (us-east-1, us-west-2) + [Configure a VPC for Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/usingVPC.html) +- **IAM fine-grained access control**: agent/alias IAM policies using resource `*` + without justification → MEDIUM. Follow least privilege for inference endpoints. + [IAM for Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/security-iam.html) +- **Knowledge Base logging**: KB ingestion log delivery not configured → LOW. + CloudTrail and CloudWatch not enabled for anomaly detection → MEDIUM. +- **Prompt injection**: prompt templates not hardened against injection → guidance. + [Prompt engineering best practices](https://docs.aws.amazon.com/prescriptive-guidance/latest/llm-prompt-engineering-best-practices/introduction.html) +- **Model access**: model access not scoped to essential models (least privilege) → + MEDIUM. + [Model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) + +### 4.2 Performance +Ref: [Monitoring Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring.html) + +- **Latency and throttling**: `InvocationLatency` p95, `TimeToFirstToken` p95, + throttle rate (`InvocationThrottles / Invocations`), and server error rate. High + throttle rate → HIGH (add retries with exponential backoff + jitter, request quota + increase, spread load, use CRIS). Prefer smaller models and streaming for + latency-sensitive apps. + [Improve Bedrock performance](https://repost.aws/knowledge-center/bedrock-improve-performance-latency) +- **Agent performance**: agents can use a latency-optimized flow when they have a + single knowledge base, no enabled action groups, don't ask follow-ups, and use the + default orchestration template → flag agents that miss these conditions. + [Optimize agent performance](https://docs.aws.amazon.com/bedrock/latest/userguide/agents-optimize-performance.html) +- **Agent version configuration drift**: compare deployed alias versions (from + `GetAgentVersion`) against the draft. Aliases pinned to outdated versions with + different model IDs or orchestration types than the current draft → MEDIUM (may + miss performance or capability improvements). +- **Invoked model versions**: models in Legacy or End-of-Life state still receiving + invocations → MEDIUM (plan upgrade; Legacy state lasts ≥6 months before EOL). + [Model lifecycle](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle.html) +- **Data automation**: success rate <95%, error rate >5%, or throttle rate >1% for + production workloads → MEDIUM. + [Bedrock Data Automation](https://docs.aws.amazon.com/bedrock/latest/userguide/bda.html) +- **Bedrock service tier**: verify tier selection (`default`, `flex`, `priority`, + `reserved`) matches workload criticality. Latency-sensitive apps on Flex, or batch + workloads on Priority, are misaligned → MEDIUM. + [Service tiers](https://docs.aws.amazon.com/bedrock/latest/userguide/service-tiers-inference.html) + +### 4.3 Service Quotas +Ref: [Bedrock quotas](https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html) + +- **Model quotas**: compare observed P95 invocations-per-minute and tokens-per-minute + against the account RPM/TPM quotas (including CRIS and Global CRIS quotas) from + `servicequotas.GetServiceQuota`. Utilization >75% → MEDIUM (request increase before + throttling); sustained near the limit → HIGH. +- **Guardrail service quotas**: track ApplyGuardrail requests and text-unit + consumption for content filter, denied topic, sensitive information, word filter, + and contextual grounding policies. Utilization >75% → MEDIUM. Note: CloudWatch + metrics don't distinguish Classic vs Standard policy versions, so review manually + when both are configured. + [Guardrail quotas](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-quotas.html) + +### 4.4 Cost Optimization +Ref: [Bedrock pricing](https://aws.amazon.com/bedrock/pricing/) + +- **Application inference profiles**: not used for cost allocation / tagging → LOW. + App inference profiles integrate with Cost Allocation Tags for usage tracking. + [Track cost and usage with inference profiles](https://aws.amazon.com/blogs/machine-learning/track-allocate-and-manage-your-generative-ai-cost-and-usage-with-amazon-bedrock/) +- **Custom model distillation**: narrow, repetitive, high-volume tasks on premium + large models are distillation candidates. Rule of thumb: `InvocationLatency` p99 + >3000 ms AND `InputTokenCount` >1M/month AND throttle rate >5% → high-priority + candidate → MEDIUM opportunity. (us-east-1, us-west-2) + [Model distillation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-distillation.html) +- **Prompt caching**: cache offload % = + `CacheReadInputTokenCount / (CacheReadInputTokenCount + CacheWriteInputTokenCount)`. + Statuses: Not Supported, Not Enabled, Misconfigured (writes but no reads), + Underutilized (<50%), Optimized (≥50%). Input-heavy repeated context that isn't + cached → MEDIUM opportunity (up to ~85% latency and ~90% cost reduction on cached + prefixes). + [Prompt caching](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html) +- **Prompt management**: presence indicates adoption maturity — INFO. Encourage + versioning and variant testing for cost/quality tradeoffs. + [Prompt management](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-management.html) +- **Intelligent prompt routing**: routes requests within a model family to the + best-quality/lowest-cost model. Absence on mixed-complexity workloads → LOW + opportunity. + [Intelligent prompt routing](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html) +- **Provisioned throughput**: balance provisioned commitments against on-demand. + Idle or expiring provisioned models, or steady baseline load on on-demand → MEDIUM. + [Provisioned Throughput](https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.html) +- **Batch inference opportunity**: high-volume (>10K/day), latency-tolerant, or + scheduled workloads on on-demand → MEDIUM (up to ~50% cost savings). Self-managed + batch on EC2 GPU / SageMaker Batch Transform → migrate to managed Batch Inference. + [Batch inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html) +- **EC2 GPU utilization** (self-managed P4/P5/P5en/P6, requires CloudWatch Agent + + NVIDIA DCGM plugin, 7-day minimum window): + - Avg GPU util <40% → over-provisioned → migrate training to Trainium2 (~30–50%). + - Avg GPU memory util >90% → OOM risk → model parallelism / larger instance. + - GPU usage CV (StdDev/Mean) >0.5 → bursty → Spot Instances / Capacity Blocks. + - Avg GPU util >80% sustained → commitment candidate → ML Savings Plans / RIs. + +### 4.5 Resilience +Ref: [Cross-region inference](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html) + +- **Cross-Region Inference (CRIS)**: CRIS adoption % = + `Profile InputTokenCount / (Profile InputTokenCount + Base Model InputTokenCount)`. + Low adoption (traffic bypassing the inference profile by invoking base models + directly) reduces burst resilience and, for global profiles, forgoes ~10% savings → + MEDIUM. + +## Step 5: Generate Report + +Generate a shareable report artifact for the review. + +Artifact naming: `bedrock-review---.md` +Example: `bedrock-review-123456789012-us-east-1-2026-04-29.md` + +Structure the Markdown document with: + +### Report Header +``` +# Amazon Bedrock Operational Review — / +Date: | Analysis window: to +Pillars reviewed: +``` + +### Executive Summary +- Health: ✅ HEALTHY / ⚠️ WARNINGS / ❌ CRITICAL +- Finding counts by severity +- Top 3 critical/high items + +### Findings by Pillar +For each of Security, Performance, Service Quotas, Cost Optimization, +Resilience: + +| # | Finding | Severity | Current State | Recommendation | + +### CloudWatch Metrics Summary +| Metric | Model | Stat | Value | Status | Finding | + +### Service Quota Utilization +| Quota | Value | Observed P95 | Utilization % | Risk | + +### Cost Optimization Opportunities +| Opportunity | Signal | Est. Impact | Effort | + +### Priority Matrix +| # | Finding | Severity | Pillar | Effort | Impact | + +### Next Steps +- Immediate (CRITICAL/HIGH — 7 days) +- Short-term (MEDIUM — 30 days) +- Long-term (LOW — 90 days) + +### Appendix — Reference Links +- [Amazon Bedrock User Guide](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) +- [Bedrock Guardrails](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html) +- [Monitoring Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring.html) +- [Bedrock Quotas](https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html) +- [Prompt Caching](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html) +- [Model Distillation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-distillation.html) +- [Batch Inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html) +- [Cross-Region Inference](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html) +- [Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html) + +## Severity Definitions + +| Severity | Definition | SLA | +|----------|------------|-----| +| CRITICAL | Immediate risk to availability, security, or data integrity | Fix within 24–48 hours | +| HIGH | Significant gap that could lead to incidents | Fix within 1 week | +| MEDIUM | Notable improvement opportunity | Plan within 30 days | +| LOW | Minor optimization or hardening | Address when convenient | +| INFO | Observation, no action required | N/A | + +## Region-Restricted Checks + +Some checks only run in specific regions: +- **VPC Configuration for Model Customization Job**: us-east-1, us-west-2 +- **Custom Model Distillation**: us-east-1, us-west-2 +- **Application Inference Profiles**: us-east-1, us-east-2, us-west-2, + ap-northeast-1, ap-northeast-2, ap-southeast-2, ap-south-1, eu-central-1, + eu-west-1, eu-west-3, us-gov-east-1, us-gov-west-1 + +Skip a check silently in unsupported regions rather than reporting a failure. + +## Known API Quirks + +- CloudWatch metrics for guardrail Content Filter and Denied Topic policies do **not** + distinguish Classic vs Standard policy versions — utilization can be inaccurate when + both are configured. Review manually. +- `cloudwatch.GetMetricData` caps at 500 metric-data queries per call — batch and + paginate for accounts with many invoked models. +- Some models are only invocable through Cross-Region Inference profiles and will show + 0 direct token usage (100% CRIS adoption by design). +- Prompt cache entries expire after ~5 minutes of inactivity; aggregate CloudWatch + windows are directional, not per-session. +- EC2 GPU signals require the CloudWatch Agent with the NVIDIA DCGM plugin; without it, + `nvidia_smi_*` metrics are absent and GPU signals can't be evaluated. + +## Data Source Boundaries + +This skill collects data exclusively through native AWS APIs +(`bedrock`, `bedrockagent`, `cloudwatch`, `servicequotas`, `ec2`). It does **not**: +- Invoke any foundation model (no data-plane calls). +- Read prompt or response content. +- Depend on any non-AWS tooling or internal scripts — the skill is self-contained on + the DevOps Agent's primary cloud-source IAM role. diff --git a/skills/bedrock-operation-review/evals/benchmark.json b/skills/bedrock-operation-review/evals/benchmark.json new file mode 100644 index 0000000..40a28fb --- /dev/null +++ b/skills/bedrock-operation-review/evals/benchmark.json @@ -0,0 +1,629 @@ +{ + "skill_name": "bedrock-operation-review", + "skill_path": "/Users/smnixon/src/tmp/tools-for-devops-agent/skills/bedrock-operation-review", + "eval_count": 5, + "runs_per_eval": 1, + "metadata": { + "timestamp": "2026-09-04T15:39:48Z" + }, + "runs": [ + { + "eval_id": "bedrock-review-smoke-test", + "run_index": 0, + "with_skill": { + "eval_id": "bedrock-review-smoke-test", + "run_index": 0, + "assertion_results": [ + { + "text": "matches regex /[a-z]{2}-[a-z]+-\\d/", + "passed": true, + "evidence": "Regex /[a-z]{2}-[a-z]+-\\d/ matched", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'model' or contains 'guardrail'", + "passed": true, + "evidence": "OR satisfied by: contains 'model' \u2014 Substring found: 'model'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'region'", + "passed": true, + "evidence": "Substring found: 'region'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'account'", + "passed": true, + "evidence": "Substring found: 'account'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + } + ], + "pass_rate": 1.0, + "summary": "With skill: 100% assertions passed", + "execution_metrics": { + "tool_calls": 1, + "token_counts": { + "input_tokens": 4, + "output_tokens": 270, + "cache_read_input_tokens": 47299, + "cache_creation_input_tokens": 28692 + } + }, + "timing": { + "elapsed_seconds": 16.217737250000937 + }, + "raw_output": "Two resources in `bedrock-context.json`:\n\n| Name | Type | Region | Account |\n|------|------|--------|---------|\n| anthropic.claude-3-5-sonnet-20241022-v2:0 | model | us-east-1 | $accountId |\n| pii-content-filter | guardrail | us-east-1 | $accountId |\n\nNote: `account` is the literal placeholder `$accountId` in the file, not a resolved account ID." + }, + "without_skill": { + "eval_id": "bedrock-review-smoke-test", + "run_index": 0, + "assertion_results": [ + { + "text": "matches regex /[a-z]{2}-[a-z]+-\\d/", + "passed": true, + "evidence": "Regex /[a-z]{2}-[a-z]+-\\d/ matched", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'model' or contains 'guardrail'", + "passed": true, + "evidence": "OR satisfied by: contains 'model' \u2014 Substring found: 'model'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'region'", + "passed": true, + "evidence": "Substring found: 'region'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'account'", + "passed": true, + "evidence": "Substring found: 'account'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + } + ], + "pass_rate": 1.0, + "summary": "Without skill: 100% assertions passed", + "execution_metrics": { + "tool_calls": 1, + "token_counts": { + "input_tokens": 4, + "output_tokens": 249, + "cache_read_input_tokens": 39119, + "cache_creation_input_tokens": 20528 + } + }, + "timing": { + "elapsed_seconds": 11.766718291997677 + }, + "raw_output": "| Name | Type | Region | Account |\n|---|---|---|---|\n| anthropic.claude-3-5-sonnet-20241022-v2:0 | model | us-east-1 | $accountId |\n| pii-content-filter | guardrail | us-east-1 | $accountId |\n\nNote: `account` is the literal placeholder `$accountId` in the file, not a resolved account ID." + }, + "delta_pass_rate": 0.0 + }, + { + "eval_id": "bedrock-review-artifact-naming", + "run_index": 0, + "with_skill": { + "eval_id": "bedrock-review-artifact-naming", + "run_index": 0, + "assertion_results": [ + { + "text": "matches regex /bedrock-review-[a-z0-9-]+-\\d{4}-\\d{2}-\\d{2}\\.md/", + "passed": false, + "evidence": "Regex /bedrock-review-[a-z0-9-]+-\\d{4}-\\d{2}-\\d{2}\\.md/ did not match", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "matches regex /[a-z]{2}-[a-z]+-\\d/", + "passed": true, + "evidence": "Regex /[a-z]{2}-[a-z]+-\\d/ matched", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + } + ], + "pass_rate": 0.5, + "summary": "With skill: 50% assertions passed", + "execution_metrics": { + "tool_calls": 1, + "token_counts": { + "input_tokens": 4, + "output_tokens": 365, + "cache_read_input_tokens": 47333, + "cache_creation_input_tokens": 28715 + } + }, + "timing": { + "elapsed_seconds": 13.221120708010858 + }, + "raw_output": "**First resource:** `anthropic.claude-3-5-sonnet-20241022-v2:0` (type: model), region **us-east-1**.\n\n**Report artifact filename:**\n```\nbedrock-review-$accountId-us-east-1-2026-09-04.md\n```\n\nNote: the account field in the file is the literal placeholder `$accountId`, so `` isn't resolved \u2014 you'd substitute the real account ID before generating the report." + }, + "without_skill": { + "eval_id": "bedrock-review-artifact-naming", + "run_index": 0, + "assertion_results": [ + { + "text": "matches regex /bedrock-review-[a-z0-9-]+-\\d{4}-\\d{2}-\\d{2}\\.md/", + "passed": false, + "evidence": "Regex /bedrock-review-[a-z0-9-]+-\\d{4}-\\d{2}-\\d{2}\\.md/ did not match", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "matches regex /[a-z]{2}-[a-z]+-\\d/", + "passed": true, + "evidence": "Regex /[a-z]{2}-[a-z]+-\\d/ matched", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + } + ], + "pass_rate": 0.5, + "summary": "Without skill: 50% assertions passed", + "execution_metrics": { + "tool_calls": 1, + "token_counts": { + "input_tokens": 4, + "output_tokens": 199, + "cache_read_input_tokens": 39151, + "cache_creation_input_tokens": 20561 + } + }, + "timing": { + "elapsed_seconds": 12.927154333010549 + }, + "raw_output": "**First resource:** `anthropic.claude-3-5-sonnet-20241022-v2:0` (type: model) in region **us-east-1**.\n\n**Report artifact filename:** `bedrock-review-2026-09-04.md`" + }, + "delta_pass_rate": 0.0 + }, + { + "eval_id": "bedrock-review-data-source-priority", + "run_index": 0, + "with_skill": { + "eval_id": "bedrock-review-data-source-priority", + "run_index": 0, + "assertion_results": [ + { + "text": "contains 'Bedrock' or contains 'bedrock'", + "passed": true, + "evidence": "OR satisfied by: contains 'bedrock' \u2014 Substring found: 'bedrock'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'CloudWatch' or contains 'cloudwatch'", + "passed": true, + "evidence": "OR satisfied by: contains 'cloudwatch' \u2014 Substring found: 'cloudwatch'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'ListGuardrails' or contains 'GetMetricData' or contains 'ListMetrics' or contains 'GetServiceQuota'", + "passed": true, + "evidence": "OR satisfied by: contains 'listguardrails' \u2014 Substring found: 'listguardrails'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'not' or contains 'no '", + "passed": true, + "evidence": "OR satisfied by: contains 'not' \u2014 Substring found: 'not'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + } + ], + "pass_rate": 1.0, + "summary": "With skill: 100% assertions passed", + "execution_metrics": { + "tool_calls": 0, + "token_counts": { + "input_tokens": 2, + "output_tokens": 457, + "cache_read_input_tokens": 11329, + "cache_creation_input_tokens": 24685 + } + }, + "timing": { + "elapsed_seconds": 12.107662666006945 + }, + "raw_output": "The skill collects data through **AWS control-plane APIs and CloudWatch metrics only** \u2014 no data-plane calls. Specific APIs/namespaces it names:\n\n1. **`bedrock`** \u2014 e.g. `bedrock.ListFoundationModels`, `bedrock.ListGuardrails`, `bedrock.GetModelInvocationLoggingConfiguration`, `bedrock.ListProvisionedModelThroughputs`\n2. **`bedrockagent`** \u2014 e.g. `bedrockagent.ListAgents`, `bedrockagent.GetAgentVersion`, `bedrockagent.ListKnowledgeBases`\n3. **`cloudwatch`** \u2014 `cloudwatch.ListMetrics` and `cloudwatch.GetMetricData` against the `AWS/Bedrock` namespace (plus `CWAgent` for GPU metrics)\n4. **`servicequotas`** \u2014 `servicequotas.GetServiceQuota`\n5. **`ec2`** \u2014 for self-managed GPU instance context\n\n**No.** The skill explicitly does not invoke any foundation model (no data-plane calls) and does not read prompt or response content. It states this in both the intro (\"performs no data-plane model invocations and reads no prompt or response content\") and the Data Source Boundaries section." + }, + "without_skill": { + "eval_id": "bedrock-review-data-source-priority", + "run_index": 0, + "assertion_results": [ + { + "text": "contains 'Bedrock' or contains 'bedrock'", + "passed": true, + "evidence": "OR satisfied by: contains 'bedrock' \u2014 Substring found: 'bedrock'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'CloudWatch' or contains 'cloudwatch'", + "passed": false, + "evidence": "No OR branch satisfied \u2014 contains 'cloudwatch': Substring not found: 'cloudwatch'; contains 'cloudwatch': Substring not found: 'cloudwatch'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'ListGuardrails' or contains 'GetMetricData' or contains 'ListMetrics' or contains 'GetServiceQuota'", + "passed": false, + "evidence": "No OR branch satisfied \u2014 contains 'listguardrails': Substring not found: 'listguardrails'; contains 'getmetricdata': Substring not found: 'getmetricdata'; contains 'listmetrics': Substring not found: 'listmetrics'; contains 'getservicequota': Substring not found: 'getservicequota'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'not' or contains 'no '", + "passed": true, + "evidence": "OR satisfied by: contains 'no ' \u2014 Substring found: 'no '", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + } + ], + "pass_rate": 0.5, + "summary": "Without skill: 50% assertions passed", + "execution_metrics": { + "tool_calls": 1, + "token_counts": { + "input_tokens": 4, + "output_tokens": 572, + "cache_read_input_tokens": 39163, + "cache_creation_input_tokens": 20768 + } + }, + "timing": { + "elapsed_seconds": 21.110713707996183 + }, + "raw_output": "I don't see a \"Bedrock operational review\" skill in the available skills list. Let me search for it on disk.There is no \"Bedrock operational review\" skill installed. I checked the available-skills list (none match) and searched `~/.claude` on disk \u2014 the only files mentioning \"bedrock\" are an unrelated `security-guidance` plugin README, its `llm.py` hook, and a marketplace manifest, none of which is a Bedrock operational-review skill.\n\nSo I can't tell you which AWS APIs \"the skill\" uses or whether it invokes a foundation model \u2014 that skill doesn't exist here to inspect. I'd rather say that than invent API names.\n\nIf you meant a specific skill, point me at its file or exact name and I'll read it and answer precisely." + }, + "delta_pass_rate": 0.5 + }, + { + "eval_id": "bedrock-review-severity-definitions", + "run_index": 0, + "with_skill": { + "eval_id": "bedrock-review-severity-definitions", + "run_index": 0, + "assertion_results": [ + { + "text": "contains 'CRITICAL'", + "passed": true, + "evidence": "Substring found: 'critical'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'HIGH'", + "passed": true, + "evidence": "Substring found: 'high'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'MEDIUM'", + "passed": true, + "evidence": "Substring found: 'medium'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'LOW'", + "passed": true, + "evidence": "Substring found: 'low'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'INFO'", + "passed": true, + "evidence": "Substring found: 'info'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + } + ], + "pass_rate": 1.0, + "summary": "With skill: 100% assertions passed", + "execution_metrics": { + "tool_calls": 0, + "token_counts": { + "input_tokens": 2, + "output_tokens": 29, + "cache_read_input_tokens": 11329, + "cache_creation_input_tokens": 24633 + } + }, + "timing": { + "elapsed_seconds": 9.490278542012675 + }, + "raw_output": "CRITICAL, HIGH, MEDIUM, LOW, INFO." + }, + "without_skill": { + "eval_id": "bedrock-review-severity-definitions", + "run_index": 0, + "assertion_results": [ + { + "text": "contains 'CRITICAL'", + "passed": false, + "evidence": "Substring not found: 'critical'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'HIGH'", + "passed": false, + "evidence": "Substring not found: 'high'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'MEDIUM'", + "passed": false, + "evidence": "Substring not found: 'medium'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'LOW'", + "passed": false, + "evidence": "Substring not found: 'low'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'INFO'", + "passed": false, + "evidence": "Substring not found: 'info'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + } + ], + "pass_rate": 0.0, + "summary": "Without skill: 0% assertions passed", + "execution_metrics": { + "tool_calls": 0, + "token_counts": { + "input_tokens": 0, + "output_tokens": 0, + "cache_read_input_tokens": 0, + "cache_creation_input_tokens": 0 + } + }, + "timing": { + "elapsed_seconds": 120.0107425830065 + }, + "raw_output": "" + }, + "delta_pass_rate": 1.0 + }, + { + "eval_id": "bedrock-review-pillars", + "run_index": 0, + "with_skill": { + "eval_id": "bedrock-review-pillars", + "run_index": 0, + "assertion_results": [ + { + "text": "contains 'Security' or contains 'security'", + "passed": true, + "evidence": "OR satisfied by: contains 'security' \u2014 Substring found: 'security'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'Performance' or contains 'performance'", + "passed": true, + "evidence": "OR satisfied by: contains 'performance' \u2014 Substring found: 'performance'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'Quota' or contains 'quota'", + "passed": true, + "evidence": "OR satisfied by: contains 'quota' \u2014 Substring found: 'quota'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'Cost' or contains 'cost'", + "passed": true, + "evidence": "OR satisfied by: contains 'cost' \u2014 Substring found: 'cost'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'Resilience' or contains 'resilience'", + "passed": true, + "evidence": "OR satisfied by: contains 'resilience' \u2014 Substring found: 'resilience'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + } + ], + "pass_rate": 1.0, + "summary": "With skill: 100% assertions passed", + "execution_metrics": { + "tool_calls": 0, + "token_counts": { + "input_tokens": 2, + "output_tokens": 77, + "cache_read_input_tokens": 11329, + "cache_creation_input_tokens": 24632 + } + }, + "timing": { + "elapsed_seconds": 11.03785154200159 + }, + "raw_output": "The skill evaluates five pillars (mirroring the Bedrock operational review structure):\n\n1. **Security**\n2. **Performance**\n3. **Service Quotas**\n4. **Cost Optimization**\n5. **Resilience**" + }, + "without_skill": { + "eval_id": "bedrock-review-pillars", + "run_index": 0, + "assertion_results": [ + { + "text": "contains 'Security' or contains 'security'", + "passed": true, + "evidence": "OR satisfied by: contains 'security' \u2014 Substring found: 'security'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'Performance' or contains 'performance'", + "passed": true, + "evidence": "OR satisfied by: contains 'performance' \u2014 Substring found: 'performance'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'Quota' or contains 'quota'", + "passed": true, + "evidence": "OR satisfied by: contains 'quota' \u2014 Substring found: 'quota'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'Cost' or contains 'cost'", + "passed": true, + "evidence": "OR satisfied by: contains 'cost' \u2014 Substring found: 'cost'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + }, + { + "text": "contains 'Resilience' or contains 'resilience'", + "passed": true, + "evidence": "OR satisfied by: contains 'resilience' \u2014 Substring found: 'resilience'", + "method": "deterministic", + "confidence": 1.0, + "uncertain": false + } + ], + "pass_rate": 1.0, + "summary": "Without skill: 100% assertions passed", + "execution_metrics": { + "tool_calls": 7, + "token_counts": { + "input_tokens": 14, + "output_tokens": 1386, + "cache_read_input_tokens": 238103, + "cache_creation_input_tokens": 38376 + } + }, + "timing": { + "elapsed_seconds": 116.64820712499204 + }, + "raw_output": "courtThe `bedrock-operation-review` skill evaluates **five pillars** (mirroring the Bedrock operational review structure, aligned with the AWS Well-Architected Framework):\n\n1. **Security** \u2014 guardrails, invocation logging, KB config/encryption, VPC for customization jobs, IAM least privilege, model access scoping, prompt-injection hardening.\n2. **Performance** \u2014 latency/throttling, agent perf & version drift, model lifecycle (Legacy/EOL), data automation, service tier alignment.\n3. **Service Quotas** \u2014 RPM/TPM utilization vs account quotas (incl. CRIS), guardrail/text-unit quota consumption.\n4. **Cost Optimization** \u2014 inference profiles for cost allocation, model distillation, prompt caching, prompt management, intelligent prompt routing, provisioned throughput, batch inference, EC2 GPU utilization.\n5. **Resilience** \u2014 Cross-Region Inference (CRIS) adoption." + }, + "delta_pass_rate": 0.0 + } + ], + "run_summary": { + "with_skill": { + "mean_pass_rate": 0.9, + "stddev_pass_rate": 0.2236, + "mean_tokens": 239.6, + "mean_input_tokens": 2.8, + "mean_output_tokens": 239.6, + "mean_total_tokens": 242.4, + "mean_tool_calls": 0.4 + }, + "without_skill": { + "mean_pass_rate": 0.6, + "stddev_pass_rate": 0.4183, + "mean_tokens": 481.2, + "mean_input_tokens": 5.2, + "mean_output_tokens": 481.2, + "mean_total_tokens": 486.4, + "mean_tool_calls": 2.0 + }, + "delta": { + "pass_rate": 0.3, + "tokens": -241.6, + "total_tokens": -244.0, + "input_tokens": -2.4, + "tool_calls": -1.6 + }, + "cost_efficiency": { + "quality_delta": 0.3, + "cost_delta_pct": -50.2, + "classification": "PARETO_BETTER", + "emoji": "\ud83d\udfe2", + "description": "Skill improves quality while reducing cost" + }, + "estimated_cost": { + "with_skill_per_run": { + "input_cost": 8e-06, + "output_cost": 0.003594, + "total_cost": 0.003602, + "model": "sonnet", + "currency": "USD" + }, + "without_skill_per_run": { + "input_cost": 1.6e-05, + "output_cost": 0.007218, + "total_cost": 0.007234, + "model": "sonnet", + "currency": "USD" + }, + "per_eval_pair": 0.010836, + "total_runs": 5, + "total_cost": 0.0542, + "model": "sonnet", + "currency": "USD" + } + }, + "scores": { + "outcome": 0.9, + "process": 0.2, + "style": 0.9, + "efficiency": 1.0, + "overall": 0.75 + }, + "passed": true +} \ No newline at end of file diff --git a/skills/bedrock-operation-review/evals/eval_queries.json b/skills/bedrock-operation-review/evals/eval_queries.json new file mode 100644 index 0000000..b2a4401 --- /dev/null +++ b/skills/bedrock-operation-review/evals/eval_queries.json @@ -0,0 +1,8 @@ +[ + {"query": "Which skill would help me run a Bedrock operational review? Just name it; do not run it.", "should_trigger": true}, + {"query": "Is there a skill available for auditing Amazon Bedrock workloads against best practices? Answer yes or no with the skill name; do not execute it.", "should_trigger": true}, + {"query": "Name the skill that covers Bedrock security, cost optimization, and quota reviews. Do not run any audit.", "should_trigger": true}, + {"query": "Write a Python script that sorts a list of numbers", "should_trigger": false}, + {"query": "What's the weather forecast for Sydney this weekend?", "should_trigger": false}, + {"query": "Create a CloudFormation template for an S3 bucket", "should_trigger": false} +] diff --git a/skills/bedrock-operation-review/evals/evals.json b/skills/bedrock-operation-review/evals/evals.json new file mode 100644 index 0000000..d692bcc --- /dev/null +++ b/skills/bedrock-operation-review/evals/evals.json @@ -0,0 +1,62 @@ +[ + { + "id": "bedrock-review-smoke-test", + "prompt": "Read bedrock-context.json. List each resource's name, type (model or guardrail), region, and account. No analysis needed.", + "expected_output": "Lists every resource from files/bedrock-context.json with its name, type, region, and account exactly as defined in the file.", + "files": ["files/bedrock-context.json"], + "assertions": [ + "matches regex /[a-z]{2}-[a-z]+-\\d/", + "contains 'model' or contains 'guardrail'", + "contains 'region'", + "contains 'account'" + ] + }, + { + "id": "bedrock-review-artifact-naming", + "prompt": "Read bedrock-context.json. Identify the FIRST resource listed and its region, then state the report artifact filename you would generate for this review (use today's date in YYYY-MM-DD form). Do not run the audit.", + "expected_output": "Identifies the first resource and region from files/bedrock-context.json and proposes an artifact name of the form bedrock-review---.md.", + "files": ["files/bedrock-context.json"], + "assertions": [ + "matches regex /bedrock-review-[a-z0-9-]+-\\d{4}-\\d{2}-\\d{2}\\.md/", + "matches regex /[a-z]{2}-[a-z]+-\\d/" + ] + }, + { + "id": "bedrock-review-data-source-priority", + "prompt": "According to the skill, which AWS APIs does the Bedrock operational review use to collect data? Name at least three specific API calls or namespaces the skill mentions. Confirm whether the skill invokes any foundation model or reads prompt/response content. No account access required.", + "expected_output": "States the skill uses Bedrock, Bedrock Agent, CloudWatch, Service Quotas, and EC2 APIs and explicitly does NOT invoke any foundation model or read prompt/response content. Names specific calls such as bedrock.ListGuardrails, cloudwatch.GetMetricData, cloudwatch.ListMetrics, or servicequotas.GetServiceQuota.", + "files": [], + "assertions": [ + "contains 'Bedrock' or contains 'bedrock'", + "contains 'CloudWatch' or contains 'cloudwatch'", + "contains 'ListGuardrails' or contains 'GetMetricData' or contains 'ListMetrics' or contains 'GetServiceQuota'", + "contains 'not' or contains 'no '" + ] + }, + { + "id": "bedrock-review-severity-definitions", + "prompt": "List the five severity levels the skill uses for findings, in order from most to least urgent. No account access required.", + "expected_output": "Lists CRITICAL, HIGH, MEDIUM, LOW, INFO in that order.", + "files": [], + "assertions": [ + "contains 'CRITICAL'", + "contains 'HIGH'", + "contains 'MEDIUM'", + "contains 'LOW'", + "contains 'INFO'" + ] + }, + { + "id": "bedrock-review-pillars", + "prompt": "List the pillars the skill evaluates during a Bedrock operational review. No account access required.", + "expected_output": "Mentions Security, Performance, Service Quotas, Cost Optimization, and Resilience.", + "files": [], + "assertions": [ + "contains 'Security' or contains 'security'", + "contains 'Performance' or contains 'performance'", + "contains 'Quota' or contains 'quota'", + "contains 'Cost' or contains 'cost'", + "contains 'Resilience' or contains 'resilience'" + ] + } +] diff --git a/skills/bedrock-operation-review/evals/files/bedrock-context.json b/skills/bedrock-operation-review/evals/files/bedrock-context.json new file mode 100644 index 0000000..8aa7f9b --- /dev/null +++ b/skills/bedrock-operation-review/evals/files/bedrock-context.json @@ -0,0 +1,6 @@ +{ + "resources": [ + { "type": "model", "name": "anthropic.claude-3-5-sonnet-20241022-v2:0", "region": "us-east-1", "account": "$accountId" }, + { "type": "guardrail", "name": "pii-content-filter", "region": "us-east-1", "account": "$accountId" } + ] +} diff --git a/skills/bedrock-operation-review/evals/report.json b/skills/bedrock-operation-review/evals/report.json new file mode 100644 index 0000000..313b839 --- /dev/null +++ b/skills/bedrock-operation-review/evals/report.json @@ -0,0 +1,65 @@ +{ + "skill_name": "bedrock-operation-review", + "skill_path": "/Users/smnixon/src/tmp/tools-for-devops-agent/skills/bedrock-operation-review", + "timestamp": "2026-09-04T15:43:00Z", + "overall_score": 0.9, + "overall_grade": "A", + "passed": true, + "sections": { + "audit": { + "score": 100, + "grade": "A", + "passed": true, + "normalized": 1.0, + "critical": 0, + "warning": 0, + "info": 0 + }, + "functional": { + "overall": 0.75, + "grade": "C", + "passed": true, + "scores": { + "outcome": 0.9, + "process": 0.2, + "style": 0.9, + "efficiency": 1.0, + "overall": 0.75 + }, + "cost_efficiency": { + "quality_delta": 0.3, + "cost_delta_pct": -50.2, + "classification": "PARETO_BETTER", + "emoji": "\ud83d\udfe2", + "description": "Skill improves quality while reducing cost" + }, + "estimated_cost": { + "with_skill_per_run": { + "input_cost": 8e-06, + "output_cost": 0.003594, + "total_cost": 0.003602, + "model": "sonnet", + "currency": "USD" + }, + "without_skill_per_run": { + "input_cost": 1.6e-05, + "output_cost": 0.007218, + "total_cost": 0.007234, + "model": "sonnet", + "currency": "USD" + }, + "per_eval_pair": 0.010836, + "total_runs": 5, + "total_cost": 0.0542, + "model": "sonnet", + "currency": "USD" + } + }, + "trigger": { + "pass_rate": 1.0, + "grade": "A", + "passed": true, + "total_queries": 6 + } + } +} \ No newline at end of file diff --git a/skills/bedrock-operation-review/evals/trigger_report.json b/skills/bedrock-operation-review/evals/trigger_report.json new file mode 100644 index 0000000..865f0c5 --- /dev/null +++ b/skills/bedrock-operation-review/evals/trigger_report.json @@ -0,0 +1,94 @@ +{ + "skill_name": "bedrock-operation-review", + "skill_path": "/Users/smnixon/src/tmp/tools-for-devops-agent/skills/bedrock-operation-review", + "query_results": [ + { + "query": "Which skill would help me run a Bedrock operational review? Just name it; do not run it.", + "should_trigger": true, + "trigger_count": 3, + "run_count": 3, + "trigger_rate": 1.0, + "passed": true, + "mean_input_tokens": 2.0, + "mean_output_tokens": 17.0, + "mean_total_tokens": 19.0 + }, + { + "query": "Is there a skill available for auditing Amazon Bedrock workloads against best practices? Answer yes or no with the skill name; do not execute it.", + "should_trigger": true, + "trigger_count": 3, + "run_count": 3, + "trigger_rate": 1.0, + "passed": true, + "mean_input_tokens": 2.0, + "mean_output_tokens": 18.0, + "mean_total_tokens": 20.0 + }, + { + "query": "Name the skill that covers Bedrock security, cost optimization, and quota reviews. Do not run any audit.", + "should_trigger": true, + "trigger_count": 3, + "run_count": 3, + "trigger_rate": 1.0, + "passed": true, + "mean_input_tokens": 2.0, + "mean_output_tokens": 91.3, + "mean_total_tokens": 93.3 + }, + { + "query": "Write a Python script that sorts a list of numbers", + "should_trigger": false, + "trigger_count": 0, + "run_count": 3, + "trigger_rate": 0.0, + "passed": true, + "mean_input_tokens": 2.7, + "mean_output_tokens": 362.7, + "mean_total_tokens": 365.3 + }, + { + "query": "What's the weather forecast for Sydney this weekend?", + "should_trigger": false, + "trigger_count": 0, + "run_count": 3, + "trigger_rate": 0.0, + "passed": true, + "mean_input_tokens": 2.0, + "mean_output_tokens": 122.3, + "mean_total_tokens": 124.3 + }, + { + "query": "Create a CloudFormation template for an S3 bucket", + "should_trigger": false, + "trigger_count": 0, + "run_count": 3, + "trigger_rate": 0.0, + "passed": true, + "mean_input_tokens": 4.0, + "mean_output_tokens": 907.7, + "mean_total_tokens": 911.7 + } + ], + "summary": { + "total_queries": 6, + "passed": 6, + "failed": 0, + "trigger_precision": 1.0, + "no_trigger_precision": 1.0, + "mean_total_tokens_per_run": 255.6, + "estimated_cost": { + "per_run": { + "input_cost": 7e-06, + "output_cost": 0.003798, + "total_cost": 0.003805, + "model": "sonnet", + "currency": "USD" + }, + "total_runs": 6, + "total_cost": 0.0228, + "model": "sonnet", + "currency": "USD" + } + }, + "passed": true +} \ No newline at end of file diff --git a/skills/bedrock-operation-review/references/best-practices-checklist.md b/skills/bedrock-operation-review/references/best-practices-checklist.md new file mode 100644 index 0000000..b272848 --- /dev/null +++ b/skills/bedrock-operation-review/references/best-practices-checklist.md @@ -0,0 +1,55 @@ +# Amazon Bedrock Best Practices Checklist + +Organized by pillar. Maps directly to the checks in `SKILL.md` Step 4. + +## Security + +- [ ] **Guardrails** configured for high-volume workloads (filter hate, insult, sexual, violence, PII; block use-case-relevant denied topics) +- [ ] **Guardrail signals** healthy — `InvocationClientErrors` < 5% of `Invocations`; `InvocationsIntervened` in the 5–15% range on active guardrails +- [ ] **Model invocation logging** configured appropriately for the data-sensitivity profile (do NOT log prompt/output for production PII/PCI/HIPAA content) +- [ ] **Knowledge Base configuration** — chunk size 200–1000 tokens, sound embedding strategy, monitoring, error handling with retry logic +- [ ] **Knowledge Base data source encryption** — customer-managed KMS key when handling sensitive content +- [ ] **VPC configuration for model customization jobs** — training data not reachable from the internet; VPC Flow Logs enabled +- [ ] **IAM fine-grained access control** — agent/alias policies scoped to specific resources; resource `*` justified and documented +- [ ] **Knowledge Base logging** — KB ingestion log delivery configured; CloudTrail + CloudWatch enabled for anomaly detection +- [ ] **Prompt injection** — prompt templates hardened per prompt-engineering best practices +- [ ] **Model access** — scoped to only the essential, legally approved models (least privilege) + +## Performance + +- [ ] **Latency & throttling** — throttle rate low; retries use exponential backoff with jitter; smaller models and streaming used for latency-sensitive apps +- [ ] Application deployed in the same region as the Bedrock endpoint to minimize network latency +- [ ] `max_tokens` set no higher than needed; lower temperature where deterministic output suffices +- [ ] **Agent performance** — latency-optimized flow eligibility met (single KB, no enabled action groups, no follow-up questions, default orchestration template) +- [ ] **Invoked model versions** — no Legacy or End-of-Life models still receiving production traffic; upgrade plan documented +- [ ] **Data automation** — success rate > 95%, error rate < 5%, throttle rate < 1% +- [ ] **Service tier** aligned to workload (Reserved/Priority for latency-sensitive; Flex for cost-tolerant batch) + +## Service Quotas + +- [ ] **Model quotas** — RPM/TPM (including CRIS and Global CRIS) utilization < 75%; increases requested proactively +- [ ] **Guardrail service quotas** — ApplyGuardrail requests and text-unit consumption < 75% of quota; Classic vs Standard policy usage reviewed manually when both configured +- [ ] CloudWatch alarms set for high quota-utilization thresholds + +## Cost Optimization + +- [ ] **Application inference profiles** used for cost allocation / usage tracking via Cost Allocation Tags +- [ ] **Custom model distillation** evaluated for narrow, repetitive, high-volume tasks on premium large models (candidate rule: p99 latency > 3000 ms + input tokens > 1M/month + throttle rate > 5%) +- [ ] **Prompt caching** enabled and optimized (cache offload ≥ 50%) for input-heavy repeated-context workloads +- [ ] **Prompt management** used for versioning and variant testing (cost/quality tradeoffs) +- [ ] **Intelligent prompt routing** used for mixed-complexity workloads within a model family +- [ ] **Provisioned throughput** balanced against on-demand; no idle or soon-to-expire provisioned models on unused workloads +- [ ] **Batch inference** used for high-volume (> 10K/day), latency-tolerant, or scheduled workloads (up to ~50% savings); self-managed batch migrated to managed Batch Inference where models are available +- [ ] **EC2 GPU utilization** (self-managed) — no sustained under-utilization (< 40%), memory pressure (> 90%), bursty usage (CV > 0.5), or un-committed high sustained usage (> 80%) + +## Resilience + +- [ ] **Cross-Region Inference (CRIS)** adopted where burst resilience or data-residency matters; adoption % high (traffic routed through inference profiles, not direct base-model invocations) + +## CloudWatch Alarm Coverage (recommended) + +- [ ] Throttling — `InvocationThrottles` alarm per high-value model +- [ ] Latency — `InvocationLatency` p95 threshold alarm +- [ ] Server errors — `InvocationServerErrors` alarm +- [ ] Guardrail interventions — `InvocationsIntervened` trend alarm +- [ ] Quota utilization — alarm approaching RPM/TPM limits diff --git a/skills/bedrock-operation-review/references/metrics-thresholds.md b/skills/bedrock-operation-review/references/metrics-thresholds.md new file mode 100644 index 0000000..ed782d6 --- /dev/null +++ b/skills/bedrock-operation-review/references/metrics-thresholds.md @@ -0,0 +1,83 @@ +# Amazon Bedrock CloudWatch Metrics Thresholds Reference + +Metrics are retrieved via `cloudwatch.ListMetrics` (to discover invoked models) and +`cloudwatch.GetMetricData` over the selected window (default 7 days). Severity reflects +sustained values, not single spikes. `cloudwatch.GetMetricData` allows up to 500 +metric-data queries per call — batch and paginate for accounts with many models. + +## Bedrock Model Metrics (Namespace: `AWS/Bedrock`, Dimension: `ModelId`) + +| Metric | Stat | Normal | Warning | Critical | Finding | +|---|---|---|---|---|---| +| Invocations | Sum | baseline | — | — | Volume; denominator for rate calculations | +| InvocationThrottles | Sum | throttle rate ~0% | > 1% of Invocations | > 5% of Invocations | Add retries w/ backoff+jitter, request quota increase, use CRIS | +| InvocationServerErrors | Sum | ~0% | > 2% of Invocations | > 5% of Invocations | Investigate service-side failures / problematic content | +| InvocationClientErrors | Sum | < 1% | > 5% of Invocations | > 15% of Invocations | Bad or blocked input reaching models; add guardrails | +| InvocationLatency | Average | task-dependent | rising trend | sustained high | Use smaller model / streaming; deploy in-region | +| InvocationLatency | p95 | task-dependent | > 2000 ms | > 3000 ms | Latency-sensitive UX risk; distillation candidate | +| TimeToFirstToken | Average, p95 | low | rising | high | Perceived latency — prefer streaming | +| InputTokenCount | Sum | baseline | high | very high | Input-heavy = prompt-caching / distillation candidate | +| OutputTokenCount | Sum | baseline | — | — | Output volume; cost driver | +| InvocationsIntervened | Sum | 5–15% of Invocations | < 5% (under-config) | > 15% (over-tuned) | Review guardrail policy configuration | +| TextUnitCount | Sum | tracked | approaching quota | at quota | Guardrail consumption maps to billing + quota | +| CacheReadInputTokenCount | Sum | present | writes without reads | — | Prompt-cache reads; used for offload % | +| CacheWriteInputTokenCount | Sum | present | high vs reads | — | Prompt-cache writes; used for offload % | + +**Derived rates:** +- Throttle rate = `InvocationThrottles / Invocations` +- Server error rate = `InvocationServerErrors / Invocations` +- Client error rate = `InvocationClientErrors / Invocations` +- Intervened rate = `InvocationsIntervened / Invocations` +- Cache offload % = `CacheReadInputTokenCount / (CacheReadInputTokenCount + CacheWriteInputTokenCount) * 100` +- CRIS adoption % = `Profile InputTokenCount / (Profile InputTokenCount + Base Model InputTokenCount) * 100` + +## Prompt Caching Status (derived) + +| Status | Condition | +|---|---| +| Not Supported | Model not eligible for prompt caching | +| Not Enabled | Caching supported but no cache activity detected | +| Misconfigured | Cache writes occurring but no cache reads (prompts may not share a common prefix) | +| Underutilized | Cache offload % below 50% | +| Optimized | Cache offload % 50% or higher | + +`cachingPotential` from output/input ratio: High (< 0.2), Medium (0.2–0.5), Low (> 0.5). + +## Service Quota Utilization + +| Quota family | Source | Warning | Critical | Finding | +|---|---|---|---|---| +| Model RPM / CRIS RPM / Global CRIS RPM | `servicequotas.GetServiceQuota` vs observed P95 invocations/min | > 75% | sustained near 100% | Request increase; spread load; adopt CRIS | +| Model TPM / CRIS TPM / Global CRIS TPM | `servicequotas.GetServiceQuota` vs observed P95 tokens/min | > 75% | sustained near 100% | Request increase; optimize token usage | +| Guardrail ApplyGuardrail + policy text units | quota vs text-unit consumption | > 75% | near 100% | Request increase; optimize policies; distribute regionally | + +> Guardrail CloudWatch metrics do not distinguish Classic vs Standard policy versions. +> When both are configured, utilization can be inaccurate — review manually. + +## EC2 GPU Utilization (Namespace: `CWAgent`, NVIDIA DCGM plugin, 7-day min window) + +| Signal | Metric | Threshold | Recommendation | +|---|---|---|---| +| Low GPU utilization | `nvidia_smi_utilization_gpu` avg | < 40% | Migrate training to Trainium2 (~30–50% savings) | +| High GPU memory pressure | `nvidia_smi_memory_util` avg | > 90% | Model/tensor parallelism or larger instance (p5en/p6) | +| Bursty / intermittent usage | CV = StdDev/Mean of GPU util | > 0.5 | EC2 Spot Instances (up to 90%) / Capacity Blocks | +| High sustained utilization | `nvidia_smi_utilization_gpu` avg | > 80% | ML Savings Plans / Reserved Instances (up to 64%) | + +If "CloudWatch Agent Installed" = No, `nvidia_smi_*` metrics are absent and GPU signals cannot be evaluated. + +## Data Automation + +| Metric | Production target | Finding | +|---|---|---| +| Success rate | > 95% | Below target → investigate IAM/input/limits | +| Error rate | < 5% | Above → check permissions and input format | +| Throttle rate | < 1% | Above → client-side rate limiting / quota increase | +| Average latency | baseline | Degradation → reduce input size, use async | + +## Model Lifecycle (Invoked Model Versions) + +| State | Meaning | Action | +|---|---|---| +| Active | Fully supported | No action | +| Legacy | Retirement announced; ≥ 6 months before EOL | MEDIUM — plan upgrade, test new version in non-prod | +| End-of-Life | No longer available | HIGH — migrate immediately |