Skip to content

fix: fail closed when live broker state is unreadable - #59

Closed
albertovincenzi wants to merge 1 commit into
fix/validate-breaker-recordsfrom
fix/fail-closed-live-state
Closed

fix: fail closed when live broker state is unreadable#59
albertovincenzi wants to merge 1 commit into
fix/validate-breaker-recordsfrom
fix/fail-closed-live-state

Conversation

@albertovincenzi

Copy link
Copy Markdown
Collaborator

Problem

Every live-state read used unwrap_or_default or ok, turning a broker/KV failure into valid-looking empty data. During an outage Gate could report counters at zero, no held breaker, state flowing, and an immediate ETA even though the source of truth never answered.

This affected graph detail, target lists, shared-budget views, product metrics, ETA, and recent breaches. A 200 response was therefore indistinguishable from a real unused budget.

Fix

  • Preserve errors from counter, breaker, and breaker-prefix reads.
  • Map those broker errors consistently to HTTP 502 with an explicit live-state message.
  • Make ETA and graph-detail builders return a fallible result instead of manufacturing defaults.
  • Keep a genuinely missing counter as zero; only transport/protocol failures change behavior.
  • Keep push shedding's documented best-effort behavior unchanged: if its optional preflight read fails, the relay remains authoritative.

Verification

A broker-backed live test blocks the KV surface and checks graph detail, ETA, target list, shared budgets, recent breaches, and product metrics. Every endpoint must return 502 instead of a zero/empty 200.

  • cargo test --workspace
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo fmt --all -- --check

Review / dependency

This PR is stacked on #58 because breaker reads become fallible after the record validation is centralized there. Review commit 91b69aa relative to fix/validate-breaker-records. Merge order: #44, #58, then this PR.

#30 independently bounds the recent-breaches limit and touches one nearby line; after #30 merges this branch needs only a mechanical one-line rebase preserving both changes.

@alice-viola

Copy link
Copy Markdown
Contributor

Landed on master via #67 (merge commit 944ee9b) as part of the 62-PR integration — this PR's head commit 91b69aa is an ancestor of master. GitHub could not mark it merged automatically because its base is fix/validate-breaker-records, not master. Closing as landed.

@alice-viola alice-viola closed this Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants