Expose UFFD pager stats as Prometheus metrics#297
Conversation
Wire the pager's existing atomic counters into hypeman's OTel Prometheus bridge so otel-collector can scrape them alongside the other host exporters. The JSON /stats endpoint over the control unix socket is unchanged; a new opt-in TCP /metrics endpoint reads the same snapshot. --metrics-addr enables the pull endpoint. --otel-endpoint enables OTLP push. Both are opt-in so existing deployments keep working. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pair with the new --metrics-addr flag: pass through an env var so per-instance overrides written via EnvironmentFile can opt in without editing the base unit. Empty default keeps the metrics server off for hosts that haven't been configured. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Required by scripts/check-uffd-version.sh whenever runtime files under lib/uffdpager change. The new metrics endpoint is additive and does not alter the UFFD wire protocol, but the pre-commit contract asks for an explicit version bump on any pager code change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Nobody's pushing OTLP from the pager today and there's no roadmap ask for it, so the three flags were speculative surface that confused reviewers. hypotel.Init with an empty endpoint still gives us the Prometheus pull path we actually use. Re-add when someone needs push. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ListenAndServe inside a goroutine turns bind failures (invalid address, port already in use) into a silent log line while startMetrics reports success. Split the two steps: bind up front so a bad --metrics-addr fails the pager at startup, then Serve on the already-bound listener in the goroutine. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ed934ed. Configure here.
| Environment="HYPEMAN_UFFD_METRICS_ADDR=" | ||
| EnvironmentFile=-/run/hypeman/uffd/%i.env | ||
| ExecStart=/bin/sh -c 'exec "\${HYPEMAN_UFFD_BINARY}" --data-dir "\${HYPEMAN_UFFD_DATA_DIR}" --version-key "\${HYPEMAN_UFFD_VERSION_KEY}" --cache-max-bytes "\${HYPEMAN_UFFD_CACHE_MAX_BYTES}"' | ||
| ExecStart=/bin/sh -c 'exec "\${HYPEMAN_UFFD_BINARY}" --data-dir "\${HYPEMAN_UFFD_DATA_DIR}" --version-key "\${HYPEMAN_UFFD_VERSION_KEY}" --cache-max-bytes "\${HYPEMAN_UFFD_CACHE_MAX_BYTES}" --metrics-addr "\${HYPEMAN_UFFD_METRICS_ADDR}"' |
There was a problem hiding this comment.
Metrics addr omitted from env
Medium Severity
The unit passes --metrics-addr from HYPEMAN_UFFD_METRICS_ADDR via EnvironmentFile=/run/hypeman/uffd/%i.env, but the supervisor still overwrites that file with only cache, data-dir, and version fields. The variable is never written there, so enabling metrics through the same env-file path used for other pager settings does not work without a separate unit override.
Reviewed by Cursor Bugbot for commit ed934ed. Configure here.
There was a problem hiding this comment.
Intentional — the two override paths carry different kinds of knobs.
/run/hypeman/uffd/%i.envis written by hypeman-api at snapshot restore time. It's the right place for values that vary per-session (cache size, data dir, version key).HYPEMAN_UFFD_METRICS_ADDRis set once at deploy time and doesn't change per session. It flows through a systemd drop-in written by Ansible (hypeman-uffd@<version>.service.d/metrics.conf), which is the sibling PR in kernel/infra.
We could unify by teaching hypeman-api to also write the metrics addr into the env file, but that would mean coupling the API service to the deploy-time port map, which feels backwards. Keeping the split so each owner controls its own set of overrides.
Binding /metrics is auxiliary to serving UFFD sessions. Failing the whole pager on a bind error (bad addr, port collision) starves UFFD restore for that snapshot version, which is worse than a missing Prometheus endpoint. Bind still runs synchronously so the failure is loud at startup, but the pager now logs and continues instead of exiting. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>


Summary
/metricsPrometheus endpoint tohypeman-uffd-pager, wired through the existinglib/otelOTel→Prometheus bridge already used byhypeman-api.SnapshotStats()/SnapshotTimingStats(). No new state, no changes to counter increment sites./statshandler is refactored to share the same snapshot via a new(*server).stats()method, so the JSON and Prometheus outputs stay in sync.version_keyattribute so multiple pager versions running side-by-side stay distinguishable in queries.One new flag on
hypeman-uffd-pager, opt-in:--metrics-addr— TCP address for Prometheus/metrics. Empty (the default) disables the metrics server entirely; pager behavior is unchanged for hosts that don't set it.Metrics exposed
Counters:
hypeman_uffd_{cache_hits,cache_misses,faults,backing_bytes_read,copies,copy_errors}_total, plus_nanos_totalaccumulators for cache lookup / add / fault / read-page / backing-read / copy latencies.Gauges:
hypeman_uffd_{cache_bytes,cache_max_bytes,cache_items,cache_shards,active_sessions,active_faults,max_concurrent_faults,draining}, plus high-water_max_nanosgauges for the same latencies.VERSION bump
Bumps
lib/uffdpager/VERSIONfrom 0.1.3 → 0.1.4 to satisfyscripts/check-uffd-version.sh. The check is mechanical: any change to files underlib/uffdpager/(excluding tests, README, and VERSION itself) demands a version bump. The new metrics endpoint is additive and does not alter the UFFD wire protocol, but the check is intentionally strict, so an explicit bump is the right call.Test plan
GOOS=linux go build ./cmd/uffd-pager/ ./lib/uffdpager/GOOS=linux go vet ./cmd/uffd-pager/ ./lib/uffdpager/GOOS=linux go test ./lib/uffdpager/...— includes newTestRegisterMetricsObservesStatsandTestRegisterMetricsNilMeterdev-yul-hypeman-0, curlhttp://127.0.0.1:<port>/metrics, verify Prometheus text and non-zero values on a running pagerNote
Low Risk
Observability-only: no changes to fault handling or UFFD protocol; metrics are off unless an address is configured, and bind failures log and continue without metrics.
Overview
Adds an opt-in Prometheus
/metricsendpoint tohypeman-uffd-pagervia--metrics-addr(empty default leaves behavior unchanged). When enabled, the pager initializeslib/oteland serves metrics on TCP while registering observable OTel instruments that read existing pager state through a new(*server).stats()snapshot.RegisterMetrics(linux) maps cache, fault, copy, and latency fields tohypeman_uffd_*counters and gauges, each tagged withversion_key. JSON/statsnow calls the samestats()helper so Prometheus and HTTP JSON stay aligned.Install wiring adds
HYPEMAN_UFFD_METRICS_ADDRto the systemd uffd template.lib/uffdpager/VERSIONbumps 0.1.3 → 0.1.4. Tests cover metric registration and nil-meter no-ops.Reviewed by Cursor Bugbot for commit cf5ef6b. Bugbot is set up for automated code reviews on this repo. Configure here.