Add webhook alerting to Release Health Monitor (org:release-failure-alerting)#19
Add webhook alerting to Release Health Monitor (org:release-failure-alerting)#19kayodebristol wants to merge 2 commits into
Conversation
…ailure-alerting) Detection (release-health-monitor.yml) already scans daily and reports into a single tracking issue (#18), but nothing actively notifies anyone when it fires -- the issue is silently re-edited in place. Add: - docs/adr/ADR-0001-release-health-alerting.md: design decision for closing the detection-vs-alerting gap via a webhook notification. - New 'Alert webhook on tripwire' step: POSTs a Slack/Teams-compatible JSON payload to secrets.RELEASE_HEALTH_WEBHOOK_URL whenever stale_count != 0 or failed_count != 0. No-op (non-fatal) if the secret isn't configured yet, so this doesn't regress existing behavior. Closes the alerting gap in org:release-failure-alerting.
There was a problem hiding this comment.
Pull request overview
Adds active alerting to the existing Release Health Monitor so daily tripwire detections (failed/stale release state) notify a human channel via a configurable webhook, rather than only updating an issue in place.
Changes:
- Adds a new “Alert webhook on tripwire” step to post a JSON payload to
RELEASE_HEALTH_WEBHOOK_URLwhenstale_count != 0 || failed_count != 0. - Documents the design rationale and tradeoffs in a new ADR (
ADR-0001).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| docs/adr/ADR-0001-release-health-alerting.md | Documents the decision to add webhook-based alerting for tripwire-positive scans. |
| .github/workflows/release-health-monitor.yml | Adds a webhook notification step that runs on unhealthy scan results. |
| issue_number=$(gh issue list --repo ${{ github.repository }} --state open --label release-health --json number --jq '.[0].number' 2>/dev/null || echo "") | ||
| issue_url="https://github.com/${{ github.repository }}/issues/${issue_number}" |
There was a problem hiding this comment.
Fixed in 23796c6: the Alert webhook on tripwire step now guards against the literal string \null\ the same way the other two call sites already did -- \if [ -z \\ ] || [ \\ = \null\ ]\ falls back to a label-search URL instead of embedding a bogus /issues/null link.
| failed="${{ steps.scan.outputs.failed_count }}" | ||
| stale="${{ steps.scan.outputs.stale_count }}" | ||
| text=":rotating_light: Release health tripwire: *${failed}* FAILED, *${stale}* STALE release-bearing repos. Details: ${issue_url}" | ||
| payload=$(printf '{"text":"%s"}' "$text") |
There was a problem hiding this comment.
Fixed in 23796c6: the payload was genuinely missing repo names (only a text summary + issue link). Now the webhook step parses report.md for the per-repo FAILED/STALE lines and includes them as failed_repos/stale_repos fields plus a human-readable text field, so the alert is actionable without opening the tracking issue, matching the ADR-0001 claim.
| stale="${{ steps.scan.outputs.stale_count }}" | ||
| text=":rotating_light: Release health tripwire: *${failed}* FAILED, *${stale}* STALE release-bearing repos. Details: ${issue_url}" | ||
| payload=$(printf '{"text":"%s"}' "$text") | ||
| curl -sS -X POST -H 'Content-Type: application/json' -d "$payload" "$WEBHOOK_URL" || echo "Webhook POST failed (non-fatal)." |
There was a problem hiding this comment.
Fixed in 23796c6: added --connect-timeout 10 --max-time 30 to the curl call so a stalled webhook endpoint can no longer hang the workflow indefinitely.
…th repo names, add curl timeouts - gh issue list --jq '.[0].number' can return the literal string "null" when no issue matches; the webhook step lacked the same guard the other two call sites already had. Added. - Webhook payload now includes structured failed_repos/stale_repos (parsed from report.md) plus failed_count/stale_count/issue_url, so alerts are actionable without opening the tracking issue, matching the ADR-0001 claim. - curl now sets --connect-timeout 10 --max-time 30 so a stalled webhook endpoint cannot hang the workflow indefinitely.
Gap closed
Release-health-monitor.yml already detects failures/staleness daily and updates a tracking issue (#18), but nothing actively alerts anyone -- the issue is silently re-edited in place with no notification, which is how a repo can stay broken for months unnoticed.
This PR adds an explicit notification step:
Follow-up (not in scope here)
Relates to issue #18 (currently 4 FAILED + 20 STALE) and epic org:release-failure-alerting.