Verify production alerts against connected Azure Monitor resources

[placerda]

Problem

Cockpit's Alerts wired readiness item does not verify Azure Monitor alerting. It currently becomes ready when it finds either:

• an observability.alerts value in raw agentops.yaml, although alerts is not a valid field in the current ObservabilityConfig; or
• a textual metricAlerts / scheduledQueryRules marker in Bicep, Terraform, JSON, or YAML under a few repository paths.

This produces both false positives and false negatives:

• any unrelated alert resource can make the item green;
• disabled rules, wrong scopes, missing Action Groups, and invalid conditions are not detected;
• alert rules created directly in Azure are invisible;
• a real Foundry continuous-evaluation quality or safety alert is not recognized unless matching IaC happens to be present locally.

AgentOps already discovers the Foundry project's Application Insights resource, resolves its linked Log Analytics workspace, queries production telemetry, and constructs a read-only MonitorManagementClient. The missing capability is correlating actual Azure Monitor rules with those discovered telemetry resources.

There is also no effective scheduled fallback today: the generated daily Doctor workflow captures agentops doctor's exit code under set +e but never propagates it, so a critical regression finding can leave the workflow green.

Expected behavior

Replace static marker detection with read-only verification of real production alert configuration:

1. Resolve the Foundry project to its connected Application Insights resource and linked Log Analytics workspace using existing discovery code.
2. Inventory Microsoft.Insights/metricAlerts and Microsoft.Insights/scheduledQueryRules through Azure Monitor management APIs or Azure Resource Graph.
3. Select rules scoped to the discovered Application Insights resource or workspace.
4. Validate that matching rules are enabled and have meaningful conditions, evaluation frequency/window, thresholds, and at least one Action Group.
5. Inspect Action Group configuration without exposing receiver addresses or other sensitive values.
6. Classify relevant coverage separately for quality/drift, safety, errors, and latency instead of treating any alert as sufficient.
7. Correlate quality and safety rules with continuous-evaluation telemetry when that telemetry is available. Distinguish a correctly configured rule with no recent signal from a missing or invalid rule.
8. Keep Cockpit and Doctor read-only. Do not invoke Action Group tests or deliberately fire notifications.
9. Treat repository IaC detection as provenance/information only, not as proof that production alerts are deployed.
10. Fix the generated scheduled Doctor workflow so its final result propagates exit code 2 after artifacts and summaries are published. Document that a failed CI workflow is a fallback signal, not equivalent to an Azure Monitor operational alert.

Readiness states

• Ready: at least one relevant enabled rule targets the connected telemetry resource, has a valid condition and Action Group, and the required signal is available.
• Configured, no recent signal: the rule is valid but no corresponding telemetry was observed in the lookback window.
• Not configured: relevant telemetry exists but no matching alert rule was found.
• Misconfigured: a matching rule is disabled, points to the wrong scope, lacks actions, or has an unusable condition.
• Cannot verify: discovery, RBAC, SDK, or Azure API access prevents verification. This must not be reported as absence.
• Not applicable/hidden: the workspace is not initialized or has no discoverable production telemetry context.

Permissions and safety

Document the minimum read-only RBAC required to inventory alert rules, Action Groups, Application Insights, and Log Analytics telemetry. Verification must not require write access and must not send email, Teams, SMS, webhook, or other test notifications.

Acceptance criteria

• Alerts wired no longer becomes ready from raw YAML or textual IaC markers alone.
• The invalid observability.alerts detection path is removed.
• Existing Foundry → Application Insights → Log Analytics discovery is reused rather than duplicated.
• Real metric and scheduled-query alert rules are inventoried from Azure.
• Rules are correlated to the correct App Insights/workspace scope.
• Enabled state, condition, window/frequency, threshold, and Action Group linkage are validated.
• Alert coverage is reported by signal type: quality/drift, safety, errors, and latency.
• Cloud-side rules created outside the repository are recognized.
• IaC remains visible only as deployment provenance.
• Missing access is reported as cannot verify, never as not configured.
• No receiver secrets or addresses are written to reports, evidence, logs, or Cockpit responses.
• No notification is fired during Doctor or Cockpit verification.
• The generated daily Doctor workflow publishes its artifacts and summary, then exits with the captured Doctor exit code.
• Unit tests cover ready, no-signal, missing, disabled, wrong-scope, missing-action, inaccessible, and IaC-only cases.
• Workflow template tests prove that Doctor exit code 2 fails the scheduled job after artifact publication.
• Documentation explains the distinction between verified Azure Monitor alerting, Foundry continuous-evaluation telemetry, and CI workflow failure notifications.

[placerda]

Implemented in 66426ae (Part A: Azure Monitor alert verification) and cb5b82e (Part B: watchdog exit code).

Merged into develop via #460 (merge commit 88a89aa). This issue stays open until develop is promoted to main, where the Closes keyword will fire.