Skip to content

[8.6] Pre-fork alerting rules and operator dashboard doc #337

Description

@rootwarp

Plan reference: plan/glamsterdam-2026-09-05/issues/08-phase-8-network-soak-and-observability.md · Issue 8.6

Description

Ship a docs/gloas-observability.md note with the pre-fork alerting rules and dashboard panels over the six families this phase makes operator-visible, written last so it documents the final set.

Implementation Notes

  • The six families are fixed by D5: rvc_fork_current_id, rvc_fork_next_activation_epoch (8.1), rvc_ptc_duties_total, rvc_ptc_attestations_total (7.6, verified by 8.2), rvc_bn_capability_state (6.7, verified by 8.3), rvc_signer_rejections_total (8.4).
  • Rules, each with a stated reason: absent(rvc_fork_next_activation_epoch) while the fork is unscheduled (informational, not paging); rvc_fork_next_activation_epoch - <current epoch> < N as the pre-fork warning; PTC submission rate below 0.99 over 1 h; any rvc_bn_capability_state == 0; any increase in rvc_signer_rejections_total; any increase in rvc_validators_slashed_total (page immediately).
  • State the omit-not-zero rule explicitly, mirroring docs/validator-perf.md: an absent series means unavailable, so alerts use absent() rather than comparing against 0.
  • Cross-link Phase 7's docs/gloas-upgrade.md (7.11) and repeat that no minimum remote-signer version is writable.
  • Every backticked workspace path must resolve on disk — crates/architecture-tests/tests/docs_freshness.rs scans tracked docs/ markdown and its exemption list is shrinking-only.

Acceptance Criteria

  • docs/gloas-observability.md exists and names all six families above with its alert expression each.
  • cargo test -p rvc-architecture-tests --test docs_freshness green, no new STALE_DOC_EXEMPTIONS entry (the list asserts exactly one entry, so a second one fails outright). The doc must be git added — the scan is git ls-files -- docs, so an untracked file is silently unscanned.
  • Each rule states its firing condition and the operator action; none compares an omitted series against 0.
  • The doc records that no minimum remote-signer version is assertable (Web3Signer through 26.7.0) and that no alert or runbook step names a fork date.

Testing Notes

doc-only; the CI gate is docs_freshness. Alert expressions are validated by hand against a local scrape from 8.7's snapshot.

Points: 2 · Type: chore · Priority: P1 · Scope: 1 day · Blocked by: #332#336 · Blocks: #338

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Priority P1choreChore / maintenance

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions