Skip to content

feat: Add control plane telemetry - scheduler eligible workers histogram - #682

Open
Angela (Angelawork) wants to merge 4 commits into
agent-substrate:mainfrom
Angelawork:feature/eligible-workers-telemetry
Open

feat: Add control plane telemetry - scheduler eligible workers histogram#682
Angela (Angelawork) wants to merge 4 commits into
agent-substrate:mainfrom
Angelawork:feature/eligible-workers-telemetry

Conversation

@Angelawork

Copy link
Copy Markdown
Collaborator

Fixes #564 (Part 3)

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Description

This PR implements Part 3 of #564 by adding telemetry histogram instrumentation for ate.scheduler.eligible_workers in ateapi. It measures unassigned free worker capacity remaining after all scheduling constraint filters are applied, sampled at every scheduling decision.

Key Changes:

  • Updated Scheduler.Schedule() to record eligible candidate workers per pool (recordEligibleWorkers).
    Defined SchedulingConstraintKey (ate.scheduling.constraint) and constraint classification values (none, required_nodes, selector) in internal/ateattr/ateattr.go.
  • Added unit tests in scheduling_test.go covering candidate counts, namespaced attributes, zero-capacity fleet states, empty fleets, sandbox class mismatches, draining workers, and constraint classifications.

Testing

  • go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...
  • go test -buildvcs=false ./cmd/atenet/internal/router/...
  • make test

E2E Test

  • ./hack/create-kind-cluster.sh
  • ./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter
  • ./hack/run-e2e.sh ./internal/e2e/suites/metrics/...

@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from 5b29a8f to 1ff8de8 Compare August 4, 2026 15:52
@Angelawork
Angela (Angelawork) marked this pull request as draft August 4, 2026 17:51
@JeffLuoo

Copy link
Copy Markdown
Collaborator

Please rebase and fix e2e tests

@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch 2 times, most recently from 2cc4a44 to 59434d7 Compare August 4, 2026 18:23
@Angelawork
Angela (Angelawork) marked this pull request as ready for review August 4, 2026 18:23
@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from ad9eac4 to 74dce74 Compare August 4, 2026 18:27
@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from 74dce74 to 8f82992 Compare August 4, 2026 18:29
Comment thread internal/e2e/suites/metrics/metrics_test.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: More system metrics for debuggability (Activation SLI, crash accounting, and capacity signals)

2 participants