Skip to content

Add infrastructure, queue, and blockchain gauges to Prometheus metrics (BE-016 — partial) - #336

Merged
dDevAhmed merged 2 commits into
DigiNodes:mainfrom
annacodervibe:feat/be-016-monitoring-api
Sep 5, 2026
Merged

Add infrastructure, queue, and blockchain gauges to Prometheus metrics (BE-016 — partial)#336
dDevAhmed merged 2 commits into
DigiNodes:mainfrom
annacodervibe:feat/be-016-monitoring-api

Conversation

@annacodervibe

@annacodervibe annacodervibe commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Closes #288

BE-016 is a large issue (5-6 days scoped), so this covers a real, verified slice rather than the whole thing - the parts around infra, queue, and blockchain metrics that were computed but not actually exposed anywhere alertable.

Added five new Prometheus gauges: process_memory_usage_bytes, process_cpu_usage_microseconds, queue_jobs_total, blockchain_last_indexed_block, and blockchain_indexing_lag_blocks. These reuse data HealthService was already computing for the /health endpoint (memory, CPU, queue counts, blockchain state), just wired into scrapable metrics instead of only living in the JSON diagnostics response.

Also found and fixed a real bug along the way: getHealth() was calling the async collectDiagnostics() without awaiting it, so the /health response's diagnostics field was silently serializing a Promise object instead of real data.

Left out anything I couldn't verify against real infrastructure - disk/network/container/K8s metrics, DB replication internals, Grafana/Alertmanager/Datadog/OpenTelemetry configs, and historical time-series storage. Those need either real infra to test against or a team decision on storage, so I'd suggest splitting them into their own follow-up issues rather than bundling into this one.

Verification: tsc shows the same 143 pre-existing errors before and after (zero new ones, confirmed via exact-diff). metrics.service.spec.ts: 9/9 passing. Re-verified on a completely separate fresh clone with a fresh install, same results.

Copy link
Copy Markdown
Contributor

@annacodervibe review status for head 50cee680dc3141fd18b45951ab8c87fcbc8ef92e: the CI workflow is currently action_required, meaning a maintainer must authorize the fork workflow before results exist. This is not a contributor code failure, so no fix is requested yet.

Once CI runs, the PR will be reviewed for metric correctness, bounded label cardinality, safe failure behavior, and exposure of no credentials or sensitive data. Please keep the current focused diff stable until that check completes.

@dDevAhmed

Copy link
Copy Markdown
Contributor

resolve conflicts @annacodervibe

@annacodervibe

annacodervibe commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

resolve conflicts @annacodervibe

@dDevAhmed conflict resolved

@dDevAhmed
dDevAhmed merged commit 3f81ffa into DigiNodes:main Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BE-016 — Implement Protocol Health & System Monitoring API

2 participants