Skip to content

Stop reporting unknown queue depth as zero - #61

Closed
albertovincenzi wants to merge 2 commits into
fix/fail-closed-live-statefrom
fix/fail-closed-depth-state
Closed

Stop reporting unknown queue depth as zero#61
albertovincenzi wants to merge 2 commits into
fix/fail-closed-live-statefrom
fix/fail-closed-depth-state

Conversation

@albertovincenzi

Copy link
Copy Markdown
Collaborator

Problem

Queue depth is live broker state, but the cache currently turns a failed first read into an empty map and serves an expired successful value after later failures. Both are returned as current data.

That makes an unavailable depth endpoint look like backlog zero. The topology can appear empty even while work is queued, and ETA can become immediately available or too optimistic because it is calculated from a stale, lower backlog.

The JSON decoders also default missing partition names or pending values to empty strings and zero, so broker/API schema drift has the same misleading result.

Fix

  • Return a fallible depth result from queue-level and group-level cached readers.
  • Treat a queue as empty only when the always-supported queue-detail route also confirms 404.
  • Propagate transport, server, and decode errors through graph detail, ETA, target/graph lists, and product metrics; the API layer from fix: fail closed when live broker state is unreadable #59 maps them to HTTP 502.
  • Stop serving an expired depth as if it were live.
  • Cache failures for the same two-second TTL as successes, preserving the existing protection against an admin-API retry storm.
  • Keep the safe queue-level fallback for group depth on older Queen versions.
  • Validate the complete depth response shape and reject duplicate/missing partition data instead of defaulting it to zero.

Before / after

Before: a depth outage could produce HTTP 200 with backlog 0 or an unlabelled stale value.

After: fresh cached values remain available for two seconds; after expiry, an unreadable depth produces HTTP 502 until Queen answers again. A confirmed missing queue still has backlog zero.

Verification

  • cargo test --workspace
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo fmt --check
  • Unit coverage for current and legacy response decoding, including malformed shapes.
  • Live broker coverage proves a failed refresh is never zero/stale, is cached for one TTL, and makes graph detail, ETA, target/graph lists, and product metrics return 502.
  • Existing legacy-broker fallback test remains in place.

Review / dependency

This PR is intentionally based on #59, which introduces the common HTTP 502 mapping for unreadable live broker state. Review commit 59b9926 as this complete fix. It can be rebased after #59 lands.

@alice-viola

Copy link
Copy Markdown
Contributor

Landed on master via #67 (merge commit 944ee9b) as part of the 62-PR integration — this PR's head commit 67a088d is an ancestor of master. GitHub could not mark it merged automatically because its base is fix/fail-closed-live-state, not master. Closing as landed.

@alice-viola alice-viola closed this Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants