[Substrate] Cost & autoscaling — KEDA on the WorkerPool, 3-node budget

[Sam123ben]

Part of the Agent Substrate epic ⬆️. Depends on #A (budget) + #B (WorkerPool).

Goal

Keep Substrate within the 3-node budget and scale it to demand — ideally scale-to-zero when idle.

Today (grounded)

3-node GKE, autoscaling pool maxes ~5/zone but the operator constraint is "can't go beyond 3 nodes atm." Classic kagent's standing pods is exactly what we avoided. We have a KEDA pattern in mind (we wrote a nats-jetstream ScaledObject for the run-dispatch worker in docs/plans/nats-dispatch-workqueue/).

Tasks

• Decide WorkerPool sizing: min replicas (0 = scale-to-zero cold start, vs 1 = always-warm).
• KEDA ScaledObject on the WorkerPool driven by pending Actor demand (or queue depth); activation from zero on the first dispatch.
• Node budget: pin/limit WorkerPool resources so it can't crowd devai-api/sre/postgres on 3 nodes; node-pool selectors if a sandbox-capable pool is needed.
• Cost model: standing WorkerPool cost vs per-run Job cost — when is Substrate cheaper? (likely: many short agent calls = Substrate wins; rare long runs = Jobs).
• Alerts on WorkerPool saturation / Actor scheduling failures.

Open decisions

• Scale-to-zero (cold start per first call) vs warm floor (cost) — depends on #A cold-start numbers.
• Dedicated sandbox node pool (cleaner isolation/scaling) vs shared.

Acceptance

Substrate runs within the 3-node budget, scales up under load and down (toward zero) when idle, with no impact on core DevAI services.

Refs: docs/plans/nats-dispatch-workqueue/IMPLEMENTATION-PLAN.md (KEDA pattern), KAGENT-INTEGRATION.md §0 (3-node rationale); memory: NATS WorkQueue Dispatch, Agent Harness + kagent.

— Part of #69

[Sam123ben]

#70 completed with a NO-GO for the current WorkerPool runtime, so production WorkerPool autoscaling is not planned. The current worker is unbounded BestEffort, the canary is not Ready, the dedicated sandbox pool has zero nodes with autoscaling disabled, and there are no valid 5/20/50 Actor measurements from which to size KEDA or a cost model.

The supported Job runtime already has zero idle agent footprint and preserves the three-node budget. Reopen only after the #70 compatibility, isolation, readiness, and load gates pass; then size autoscaling from measured queue/Actor demand rather than an assumed signal.

[Sam123ben]

2026-08-20 acceptance audit: no production WorkerPool/KEDA sizing will be created from an unsafe runtime. Substrate 0.0.15 still runs workers as root/AppArmor Unconfined with hostPath, the default ServiceAccount, automounted API token, and broad capabilities. Without an approved hardened WorkerPool and valid 5/20/50 Actor measurements, autoscaling and cost sizing would be fabricated. The supported Job path continues to have zero idle agent footprint and stays within the node budget.

[Sam123ben]

Read-only node-pool audit (2026-08-20): sandbox-gvisor exists, but its backing managed instance group target size is 0, GKE autoscaling is not configured, and there are no workload=substrate nodes. The earlier implication that it would automatically scale 0→1 is not supported by the live configuration. No resize or autoscaling mutation was made. Valid 5/20/50 Actor measurements still need an approved runnable worker boundary and actual capacity.

[Sam123ben]

Local 5/20/50 concurrency measurement (isolated Kind/Colima, no production mutation), using the known-good gVisor ActorTemplate and an 8-replica WorkerPool:\n\n- 5 requested: 5 Running, 0 rejected; concurrent resume completed in 0.78s.\n- 20 requested: 8 Running on 8 distinct workers, 12 remained Suspended with ResourceExhausted: no free workers available; batch completed in 0.93s.\n- 50 requested: 8 Running on 8 distinct workers, 42 remained Suspended with the same capacity error; batch completed in 5.41s.\n- After each cohort, running Actors were suspended; the WorkerPool returned to 8/8 Ready.\n- Worker Pods retain 256Mi requests and 1Gi limits. The local runtime exposed no usable CPU/memory samples, so no fabricated per-Actor cost is reported.\n\nThis proves the current scheduler provides one concurrent Actor slot per WorkerPool replica and multiplexes over time, not simultaneous Actors within one worker. The production pool is configured for one replica but has zero schedulable sandbox nodes. Production Substrate v0.0.8 exposes the WorkerPool /scale subresource, but the live monitoring namespace has stale Prometheus Services with no Prometheus Pods/endpoints, so a Prometheus/KEDA parking-demand scaler would not have a valid source. The currently deployed runtime also predates the documented parking telemetry.\n\nConclusion: #78 now has the requested concurrency evidence, but the acceptance still fails. A truthful scaler requires (1) the hardened compatible runtime, (2) a real telemetry backend and parked-request demand metric, (3) a warm floor of at least one because golden-template readiness cannot bootstrap at zero, and (4) approved sandbox node capacity.

[Sam123ben]

Additional lifecycle timing from the isolated Actor acceptance run (2026-08-20):

• Cold Actor restore: 218.6 ms.
• Complete A2A request to synthetic provider response: 411.7 ms.
• Post-response checkpoint: 39.3 ms.
• Response content: The answer is 4.

This proves fast wake/respond/suspend behavior for one known-good golden Actor. It does not change the earlier 5/20/50 capacity result: current scheduling still provides one concurrent Actor per WorkerPool replica, production sandbox node-pool target size remains zero without a proven 0 -> 1 autoscaling path, and no parked-demand telemetry source is available for a truthful KEDA scaler. #78 remains open and no production capacity mutation was made.

[Sam123ben]

Cost-attribution increment deployed via PR #275 / 6c4d40c900600b7c5aca25973f46440bcffdca91: all legacy provider and Codex sandbox gateway calls now carry request-time tenant, user, run, agent, and provider headers. Concurrent-user tests prove task-local isolation, so gateway telemetry can assign usage to the correct owner instead of a static provider-only stream. Production is Healthy/Synced on main-6c4d40c.

#78 remains open: production demand autoscaling still cannot be claimed because the monitoring Services have no Prometheus endpoints and the sandbox-gvisor node pool has no configured 0→1 autoscaling path.