feat(gateway): let the gateway wake a deployed remote node - #222
Merged
Merged
Conversation
Plans #214: a deployed-but-stopped remote node becomes a gateway wake candidate (models list, topology, request-time wake), sourced from its own last status rather than a local Spinloop source, with a per-node wake override alongside the existing fleet-wide setting.
A deployed-but-stopped remote environment already knows what it serves — the same fact its status reply already carries — so the gateway now boots it and holds the request the way it does for a daemon node, instead of refusing outright. An undeployed environment still refuses, now from a fresh status read rather than a blanket refusal, since the control plane's own retry loop would otherwise hold the request for the whole wake timeout. Adds a per-node `wake` override alongside the fleet-wide setting, since waking a remote node costs money in a way a local daemon's engine does not.
Adds direct fleet.Wake tests for a node opting its own wake off under a fleet that wakes and on under one that does not, plus AnyNodeWakes and the remote-config resolver's missing-status and status-read-failure paths, which the implementation commit left exercised only indirectly.
The gateway can now wake a deployed-but-stopped kind: remote node, and a node entry can declare its own wake setting overriding the fleet-wide one.
…eply The control plane's status reply only relays what an environment serves while it is running — a stopped environment's status carries only state and address, deployed or not. remoteConfigFor was reading it anyway, so every deployed-but-stopped remote node reported no wakeable model and the orchestrator's dispatch failed with "reports no model to run item against". The stats reply reads the deploy config directly regardless of run state, so remoteConfigFor now resolves from a live stats call instead (no served name, though — the stats reply carries none). This also removes the "nothing deployed" pre-check StartWith gained alongside the original fix, since it read the same status reply and so had the identical blind spot; the gateway's own candidate matching already confirms deployment before a candidate is chosen, and Wake has exactly one call site for StartWith on a remote node.
Found running the fixed gateway against a real fleet: two requests raced to wake the same remote node half a second apart. A daemon node's control API turns a racing second start into a 409 the loser joins, but a remote environment's control plane has no equivalent — its instance lookup is eventually consistent right after a launch, so two racing wakes can each miss the other's not-yet-visible instance and each launch one, doubling the bill. Wake now coalesces concurrent callers for the same node (fleet path + node name) through a singleflight.Group, so only one actually starts anything. Once the node came up, a request routed to it failed with "Loading model": the engine's port was open but the model had not finished loading. statusFromRemote never mapped the control plane's own health check onto Ready, so a remote node's readiness was always unknown to waitReady — and waitReady itself fell back to a raw TCP probe for anything short of an explicit ReadyYes, including an explicit ReadyNo, when llama.cpp and vLLM both open their port well before they can answer a request. Both are fixed together: Ready is now populated from the control plane's healthy field, and waitReady only falls back to the probe when there is no reading at all, not when one already says not yet.
Syncs its 5 MODIFIED requirements across fleet-config, fleet-gateway and remote-node into the main specs, then moves the change to the archive.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets the gateway start a deployed-but-stopped
kind: remotenode the same way it already starts akind: daemonone, so a request the fleet file allows waking reaches a stopped remote environment instead of failing. Fixes #214.Summary
remoteNode.StartWithboots a deployed environment instead of refusing outright. An undeployed one is still refused, now from a fresh status read rather than a hardcoded message — needed because the control plane's own retry loop treats an undeployed environment's503the same as a capacity wait, which would otherwise hold a request for the whole wake timeout./v1/modelslist, its topology endpoint, and its request-time wake now consider a remote node's own last status — the environment's stored deploy config — instead of skipping every non-daemon node.wake: on/off, overriding the fleet-wide setting for that node alone — useful because waking a remote node boots a billed cloud instance, unlike a local daemon's engine.fleet.yaml'swakesetting and the gateway's wake behaviour updated to match.Implementation details
A remote node's wakeable config is resolved from its own last-read status (already gathered by the gateway's existing fan-out) rather than a local Spinloop source: the control plane's status reply already relays the environment's stored deploy config whether it's running or stopped, and that's the authoritative record of what a wake would start, not whatever a local fleet-file entry happens to point at.
The per-node wake check lives inside
Wake's own candidate loop rather than in the candidate-ordering helper (wakeable), soWouldWakekeeps reporting the node a wake would try regardless of policy — which is what lets a wake-off refusal still name the node it refused to start.Undeployed remote environments and the orchestrator's admission are unaffected — out of scope per the issue.