Skip to content

Expose UFFD pager stats as Prometheus metrics#297

Open
jarugupj wants to merge 7 commits into
mainfrom
hypeship/uffd-pager-metrics
Open

Expose UFFD pager stats as Prometheus metrics#297
jarugupj wants to merge 7 commits into
mainfrom
hypeship/uffd-pager-metrics

Conversation

@jarugupj

@jarugupj jarugupj commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add an opt-in TCP /metrics Prometheus endpoint to hypeman-uffd-pager, wired through the existing lib/otel OTel→Prometheus bridge already used by hypeman-api.
  • Instruments are observable, backed by callbacks reading the pager's existing atomic counters via SnapshotStats() / SnapshotTimingStats(). No new state, no changes to counter increment sites.
  • The JSON /stats handler is refactored to share the same snapshot via a new (*server).stats() method, so the JSON and Prometheus outputs stay in sync.
  • Every observation carries a version_key attribute so multiple pager versions running side-by-side stay distinguishable in queries.

One new flag on hypeman-uffd-pager, opt-in:

  • --metrics-addr — TCP address for Prometheus /metrics. Empty (the default) disables the metrics server entirely; pager behavior is unchanged for hosts that don't set it.

Metrics exposed

Counters: hypeman_uffd_{cache_hits,cache_misses,faults,backing_bytes_read,copies,copy_errors}_total, plus _nanos_total accumulators for cache lookup / add / fault / read-page / backing-read / copy latencies.

Gauges: hypeman_uffd_{cache_bytes,cache_max_bytes,cache_items,cache_shards,active_sessions,active_faults,max_concurrent_faults,draining}, plus high-water _max_nanos gauges for the same latencies.

VERSION bump

Bumps lib/uffdpager/VERSION from 0.1.3 → 0.1.4 to satisfy scripts/check-uffd-version.sh. The check is mechanical: any change to files under lib/uffdpager/ (excluding tests, README, and VERSION itself) demands a version bump. The new metrics endpoint is additive and does not alter the UFFD wire protocol, but the check is intentionally strict, so an explicit bump is the right call.

Test plan

  • GOOS=linux go build ./cmd/uffd-pager/ ./lib/uffdpager/
  • GOOS=linux go vet ./cmd/uffd-pager/ ./lib/uffdpager/
  • GOOS=linux go test ./lib/uffdpager/... — includes new TestRegisterMetricsObservesStats and TestRegisterMetricsNilMeter
  • Deploy to dev-yul-hypeman-0, curl http://127.0.0.1:<port>/metrics, verify Prometheus text and non-zero values on a running pager
  • Confirm metrics land in SigNoz after the corresponding infra PR ships the otel-collector scrape config

Note

Low Risk
Observability-only: no changes to fault handling or UFFD protocol; metrics are off unless an address is configured, and bind failures log and continue without metrics.

Overview
Adds an opt-in Prometheus /metrics endpoint to hypeman-uffd-pager via --metrics-addr (empty default leaves behavior unchanged). When enabled, the pager initializes lib/otel and serves metrics on TCP while registering observable OTel instruments that read existing pager state through a new (*server).stats() snapshot.

RegisterMetrics (linux) maps cache, fault, copy, and latency fields to hypeman_uffd_* counters and gauges, each tagged with version_key. JSON /stats now calls the same stats() helper so Prometheus and HTTP JSON stay aligned.

Install wiring adds HYPEMAN_UFFD_METRICS_ADDR to the systemd uffd template. lib/uffdpager/VERSION bumps 0.1.3 → 0.1.4. Tests cover metric registration and nil-meter no-ops.

Reviewed by Cursor Bugbot for commit cf5ef6b. Bugbot is set up for automated code reviews on this repo. Configure here.

jarugupj and others added 5 commits July 1, 2026 19:43
Wire the pager's existing atomic counters into hypeman's OTel Prometheus
bridge so otel-collector can scrape them alongside the other host
exporters. The JSON /stats endpoint over the control unix socket is
unchanged; a new opt-in TCP /metrics endpoint reads the same snapshot.

--metrics-addr enables the pull endpoint. --otel-endpoint enables OTLP
push. Both are opt-in so existing deployments keep working.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pair with the new --metrics-addr flag: pass through an env var so
per-instance overrides written via EnvironmentFile can opt in without
editing the base unit. Empty default keeps the metrics server off for
hosts that haven't been configured.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Required by scripts/check-uffd-version.sh whenever runtime files under
lib/uffdpager change. The new metrics endpoint is additive and does not
alter the UFFD wire protocol, but the pre-commit contract asks for an
explicit version bump on any pager code change.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Nobody's pushing OTLP from the pager today and there's no roadmap ask
for it, so the three flags were speculative surface that confused
reviewers. hypotel.Init with an empty endpoint still gives us the
Prometheus pull path we actually use. Re-add when someone needs push.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@jarugupj jarugupj marked this pull request as ready for review July 6, 2026 00:35
Comment thread lib/uffdpager/server_linux.go
ListenAndServe inside a goroutine turns bind failures (invalid address,
port already in use) into a silent log line while startMetrics reports
success. Split the two steps: bind up front so a bad --metrics-addr
fails the pager at startup, then Serve on the already-bound listener
in the goroutine.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ed934ed. Configure here.

Comment thread lib/uffdpager/server_linux.go
Comment thread scripts/install.sh
Environment="HYPEMAN_UFFD_METRICS_ADDR="
EnvironmentFile=-/run/hypeman/uffd/%i.env
ExecStart=/bin/sh -c 'exec "\${HYPEMAN_UFFD_BINARY}" --data-dir "\${HYPEMAN_UFFD_DATA_DIR}" --version-key "\${HYPEMAN_UFFD_VERSION_KEY}" --cache-max-bytes "\${HYPEMAN_UFFD_CACHE_MAX_BYTES}"'
ExecStart=/bin/sh -c 'exec "\${HYPEMAN_UFFD_BINARY}" --data-dir "\${HYPEMAN_UFFD_DATA_DIR}" --version-key "\${HYPEMAN_UFFD_VERSION_KEY}" --cache-max-bytes "\${HYPEMAN_UFFD_CACHE_MAX_BYTES}" --metrics-addr "\${HYPEMAN_UFFD_METRICS_ADDR}"'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Metrics addr omitted from env

Medium Severity

The unit passes --metrics-addr from HYPEMAN_UFFD_METRICS_ADDR via EnvironmentFile=/run/hypeman/uffd/%i.env, but the supervisor still overwrites that file with only cache, data-dir, and version fields. The variable is never written there, so enabling metrics through the same env-file path used for other pager settings does not work without a separate unit override.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit ed934ed. Configure here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentional — the two override paths carry different kinds of knobs.

  • /run/hypeman/uffd/%i.env is written by hypeman-api at snapshot restore time. It's the right place for values that vary per-session (cache size, data dir, version key).
  • HYPEMAN_UFFD_METRICS_ADDR is set once at deploy time and doesn't change per session. It flows through a systemd drop-in written by Ansible (hypeman-uffd@<version>.service.d/metrics.conf), which is the sibling PR in kernel/infra.

We could unify by teaching hypeman-api to also write the metrics addr into the env file, but that would mean coupling the API service to the deploy-time port map, which feels backwards. Keeping the split so each owner controls its own set of overrides.

Binding /metrics is auxiliary to serving UFFD sessions. Failing the
whole pager on a bind error (bad addr, port collision) starves UFFD
restore for that snapshot version, which is worse than a missing
Prometheus endpoint. Bind still runs synchronously so the failure is
loud at startup, but the pager now logs and continues instead of
exiting.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant