From 76d65fcabf79ab28765755259e7e27da2e4af10e Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Tue, 15 Sep 2026 00:47:25 +0100 Subject: [PATCH 1/3] docs(openspec): propose status and dashboard as top-level verbs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reading an engine's state is one operation the CLI spells twice: fleet status and remote status answer the same question about the same kind of thing, and since --env landed both work on the same environment and print differently. The renderers converged long ago; the commands above them did not. Proposes status and dashboard as top-level verbs over the existing target resolver, deleting the three spellings they replace and signposting each. The two facts only the cloud path rendered — the endpoint's health and address — ride on the node result rather than becoming a capability, since both are already in the reply the node receives. metrics and logs follow in their own change: they carry cloud-only flags with no daemon counterpart and need node capabilities to survive the move, which deserves its own review. --- .../.openspec.yaml | 2 + .../top-level-status-and-dashboard/design.md | 127 +++++++++++ .../proposal.md | 86 +++++++ .../specs/fleet-client/spec.md | 101 ++++++++ .../specs/fleet-config/spec.md | 115 ++++++++++ .../specs/remote-endpoint/spec.md | 215 ++++++++++++++++++ .../specs/remote-version-reporting/spec.md | 52 +++++ .../top-level-status-and-dashboard/tasks.md | 65 ++++++ 8 files changed, 763 insertions(+) create mode 100644 openspec/changes/top-level-status-and-dashboard/.openspec.yaml create mode 100644 openspec/changes/top-level-status-and-dashboard/design.md create mode 100644 openspec/changes/top-level-status-and-dashboard/proposal.md create mode 100644 openspec/changes/top-level-status-and-dashboard/specs/fleet-client/spec.md create mode 100644 openspec/changes/top-level-status-and-dashboard/specs/fleet-config/spec.md create mode 100644 openspec/changes/top-level-status-and-dashboard/specs/remote-endpoint/spec.md create mode 100644 openspec/changes/top-level-status-and-dashboard/specs/remote-version-reporting/spec.md create mode 100644 openspec/changes/top-level-status-and-dashboard/tasks.md diff --git a/openspec/changes/top-level-status-and-dashboard/.openspec.yaml b/openspec/changes/top-level-status-and-dashboard/.openspec.yaml new file mode 100644 index 00000000..a40cb63c --- /dev/null +++ b/openspec/changes/top-level-status-and-dashboard/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-09-14 diff --git a/openspec/changes/top-level-status-and-dashboard/design.md b/openspec/changes/top-level-status-and-dashboard/design.md new file mode 100644 index 00000000..c4a7dad6 --- /dev/null +++ b/openspec/changes/top-level-status-and-dashboard/design.md @@ -0,0 +1,127 @@ +## Context + +See proposal.md — Why. The implementation-relevant current state: + +- `resolveFleetTarget` (`cmd/spinloop/target.go`) already turns `--env` / + `--fleet` / the working directory into the `*fleet.Config` to act on, and + already holds the rule that the two flags are exclusive. Eight commands call + it. A top-level verb is a ninth caller, not a new mechanism. +- `renderFleetStatus` and the dashboard model (`dashboard_model.go`, + `dashboard_render.go`) are already independent of the command that opens + them: `fleetStatusCmd` is a flag block plus one fan-out call, and + `fleetDashboardCmd` a flag block plus `dashModelFor` and `runDashProgram`. +- `runRemoteStatus` (`cmd/spinloop/remote.go`) calls `remote.Status` directly + and renders a key-value block. It already shares `statusFact` with the fleet + table — the facts converged, the layout did not. +- Two facts it renders are not on `daemon.StatusResponse` and so do not + survive a fan-out: the endpoint's **health** (`resp.Healthy`) and its + **base URL**. `statusFromRemote` drops both. +- `movedTopLevelCommands` and `groupArgs`/`movedSubcommands` + (`cmd/spinloop/commands.go`) are the two existing signposts: the first for a + first word, the second for a group subcommand. + +## Goals / Non-Goals + +**Goals:** + +- One command reads a target's status, and one opens a board on it. +- Nothing an operator could see before is lost — including the two facts only + the cloud path rendered. +- The verbs are thin: the resolver and the renderers already exist, so the + commands are flag blocks. + +**Non-Goals:** + +- No `metrics` or `logs`. They carry cloud-only flags (`--cost`, + `--source`/`--since`/`--instance`) with no daemon counterpart, and need node + capabilities to survive the move. Their own change. +- No node capability interfaces here. Nothing in this change needs one — see + D2 for why health and base URL do not. +- No `remote` → `cloud` rename, no `kind: cloud`, no `-f` reversal. A rename + across every example and doc, deliberately separate. +- No change to what a status reports, how it degrades, or how the dashboard + behaves once open. + +## Decisions + +**D1: The verbs are new files registering at the root; the fleet spellings are +deleted, not aliased.** + +`status.go` and `dashboard.go` hold the commands. The fleet-scoped builders go, +and their bodies — `renderFleetStatus`, `dashModelFor`, `runDashProgram` — +stay where they are, since they are rendering rather than commands. + +Alternative: keep `fleetStatusCmd` and register the same builder at both +levels (rejected — two entries in the tree for one behaviour is the thing this +removes, and the house pattern for a moved command is a hard break that names +the replacement). + +**D2: Health and base URL ride on the status view, not on a node capability.** + +`remote status` renders two facts a daemon has no answer for. The tempting +move is a `Healther` capability beside `Keeper`. It is the wrong shape: both +facts are already on the reply `remoteNode.Status` receives and simply are not +copied across, so a capability would be a second round trip to fetch what the +first one already returned. + +`daemon.StatusResponse` gains neither — it is the daemon's published contract +and a daemon cannot answer either question. Instead `fleet.NodeResult` carries +them alongside the status it already holds, filled by `statusFromRemote` and +left empty by a daemon node. The renderer shows them when present. + +This keeps the capability interfaces for what they are for — an operation a +node kind can or cannot *perform* (`Keep`, `StartWithProgress`) — rather than +a field one kind happens to know. + +**D3: A single-node target renders as the table, not as the old key-value +block.** + +`remote status` printed a key-value block; `fleet status` prints a table. One +command cannot print both, and the table is the one that scales: it reads the +same for one target and for twelve, and it is what the dashboard's detail view +already echoes. + +The key-value block's extra facts are not lost — D2 carries them — but they +appear as columns, not as lines. This is the one visible difference for an +operator who used `remote status`, and it is stated in the proposal. + +Alternative: switch layout on node count (rejected — a command whose output +shape depends on how many things it found is harder to script against and +harder to explain than one that does not). + +**D4: Three signposts, using both existing mechanisms.** + +`fleet status` and `fleet dashboard` are group subcommands, so they go in +`movedSubcommands`, which `groupArgs` already reads. `remote status` is the +same shape, and the `remote` group gains the same `Args: groupArgs` the fleet +group has. No new mechanism. + +**D5: The no-target message is fixed here rather than left to the rename.** + +`fleet.Resolve`'s failure still offers only `--fleet` and creating a file, +though `--env` has been a target since it landed. With `status` at the top +level that message is what a new user meets first, and this change is what +makes them meet it. Fixing it here also means the spec's "names every way to +give one" requirement lands with the code that satisfies it. + +## Risks / Trade-offs + +- [An operator who scripted `remote status`'s key-value output gets a table] + → the layout change is stated in the proposal and the removed spelling names + its replacement, so the break is loud rather than silent; anything parsing + status output should be reading `--format=json` from `metrics`, which this + change does not touch. +- [Health and base URL become fields on `NodeResult` that only one node kind + fills] → the same is already true of several `daemon.StatusResponse` fields + (`Engine.Host` is filled by a cloud node and left empty by a daemon), so the + shape is established rather than new. +- [Two more top-level commands crowd the root] → they replace three + subcommands, so the total count falls; and `status` is among the words a + user tries first, which is the point of the move. + +## Migration Plan + +`spinloop fleet status` → `spinloop status`. `spinloop fleet dashboard` → +`spinloop dashboard`. `spinloop remote status` → `spinloop status --env `. +Each old spelling fails naming its replacement. Nothing is persisted or +transmitted differently; rollback is a revert. diff --git a/openspec/changes/top-level-status-and-dashboard/proposal.md b/openspec/changes/top-level-status-and-dashboard/proposal.md new file mode 100644 index 00000000..a52f34aa --- /dev/null +++ b/openspec/changes/top-level-status-and-dashboard/proposal.md @@ -0,0 +1,86 @@ +## Why + +Reading an engine's state is one operation, and the CLI spells it twice. +`spinloop fleet status` and `spinloop remote status` answer the same question +about the same kind of thing, and since `--env` landed both work on the same +environment and print differently. The renderers converged long ago — +`status_render.go` exists because someone noticed — but the commands above +them did not. + +The split makes the operator pick a command by how a target happens to be +configured rather than by what they want to know. A cloud environment is +`remote status`; the same environment named in a fleet file is `fleet status`; +the dashboard already draws both kinds side by side and does not care. + +Everything needed to collapse them is in place: a cloud environment is reached +through the same `fleet.Node` contract a daemon is, one resolver already turns +`--env`/`--fleet`/the working directory into the fleet to act on, and the +fan-out renders a mixed fleet without knowing which kind each node is. What is +left is to put the verb at the top level and delete the duplicate. + +This change moves the two read verbs that need nothing new to do it. `metrics` +and `logs` follow in their own change, because they carry cloud-only facts +(`--cost`, `--source`/`--since`/`--instance`) that have no daemon counterpart +and need node capabilities to survive the move. Splitting on that line keeps +this change mechanical and gives that design its own review. + +## What Changes + +- `status` and `dashboard` become top-level commands. Each takes its target + the way the fleet commands do — `--env `, `--fleet `, or the + working directory's `fleet.yaml` — and renders exactly as its fleet-scoped + spelling does today. +- **BREAKING** `spinloop fleet status`, `spinloop fleet dashboard` and + `spinloop remote status` are removed. Each fails naming the command that + replaced it, the way a moved command already does. +- **BREAKING** With no target resolvable — no `--env`, no `--fleet`, no + `./fleet.yaml` — the verbs fail rather than looking for an engine on this + machine. `spinloop serve` already shows the engine it runs, and a machine + that wants the full set locally runs `spinloop daemon` and names it in a + fleet file. The failure names all three ways to give a target, which today's + message does not: it offers only `--fleet` and creating a file, though + `--env` has been a target since it landed. +- What `remote status` reported that a node's status does not carry — the + endpoint's health and its address — is carried into the top-level verb, so + nothing an operator could see is lost with the command. +- `fleet` keeps `route`, `start`, `stop`, `deploy`, `metrics` and `logs`; + `remote` keeps everything else it has. Only `status` and `dashboard` move + here. + +## Capabilities + +### New Capabilities + +(None — every behaviour change lands in an existing capability.) + +### Modified Capabilities + +- `fleet-client`: fleet status and the fleet dashboard become top-level verbs + serving every target kind, keeping their rendering, their degradation + behaviour and the dashboard's keys unchanged. +- `fleet-config`: target resolution is stated for the top-level verbs as well + as the fleet group, and the failure with no target names all three ways to + give one. +- `remote-endpoint`: the `remote` group no longer has a `status` subcommand, + and what that subcommand reported — state, health, last-active — is reported + by the top-level `status` against the same environment. +- `remote-version-reporting`: the version an environment reports is read + through the top-level `status` rather than `remote status`. + +## Impact + +- `cmd/spinloop/status.go` and `dashboard.go`: the two verbs, registered at the + root, resolving through the existing `resolveFleetTarget`. +- `cmd/spinloop/fleet.go`, `fleet_dashboard.go`: lose their command wrappers, + keep their renderers and the dashboard model. +- `cmd/spinloop/remote.go`: loses `remoteStatusCmd` and `runRemoteStatus`; the + health and address facts it rendered move into the shared status view. +- `cmd/spinloop/commands.go`: the two verbs registered, the two fleet + subcommands unregistered, and the moved spellings signposted — `fleet + status`, `fleet dashboard` and `remote status` each naming their + replacement. +- `internal/fleet/config.go`: the no-target failure names `--env` too. +- `docs/commands/`: pages for the two verbs; `fleet.md` and `remote.md` point + at them. +- No change to the fleet file format, the environments registry, the control + plane, the daemon API, or the gateway. diff --git a/openspec/changes/top-level-status-and-dashboard/specs/fleet-client/spec.md b/openspec/changes/top-level-status-and-dashboard/specs/fleet-client/spec.md new file mode 100644 index 00000000..c7dbbba4 --- /dev/null +++ b/openspec/changes/top-level-status-and-dashboard/specs/fleet-client/spec.md @@ -0,0 +1,101 @@ +## MODIFIED Requirements + +### Requirement: Fleet status + +`spinloop status` SHALL query every node in the resolved target and render one row per node: the node name, its engine state (`idle`/`running`/`stopped`/`crashed`), what it is serving (runner and model when known), the spinloop version of the daemon on that node, and its reachability. Nodes SHALL be queried concurrently so the command's latency is that of the slowest reachable node, not their sum. + +The target SHALL resolve as it does for every other command that acts on a fleet — a named environment, a named fleet file, or the working directory's fleet file — so one environment, a fleet, and a fleet holding that same environment are all read by the one command. A cloud environment SHALL additionally report the endpoint's health and its address, which a daemon node does not carry; a target that reports neither SHALL omit them rather than render them empty. + +A node SHALL also report how long it has been since its engine last did work, taken from the activity its daemon tracks — "which of my nodes is doing nothing?" is a question a fleet view exists to answer, and the daemon already knows. That figure SHALL NOT be labelled in a way that collides with the `idle` engine state, which means something different. A node whose daemon reports no activity yet SHALL omit the figure rather than imply an engine has sat unused since it started. + +#### Scenario: Mixed fleet renders every node + +- **WHEN** `spinloop status` runs against a fleet of several nodes +- **THEN** the output has one row per node showing its state, version, and what it serves + +#### Scenario: A named environment is read by the same command + +- **WHEN** `spinloop status --env prod` runs +- **THEN** that environment's state, health and address are reported, and no + fleet file is required + +#### Scenario: A node reports how long since it last did work + +- **WHEN** `spinloop status` runs against a node whose daemon reports a last-active time +- **THEN** that node's row shows how long ago that was, labelled so it is not confused with the `idle` engine state + +#### Scenario: A node with no recorded activity omits the figure + +- **WHEN** a node's daemon reports no last-active time, because its engine has done no work yet +- **THEN** that node's row shows no activity figure rather than a misleading one + +#### Scenario: Version is shown per node + +- **WHEN** `spinloop status` runs against a fleet of running nodes +- **THEN** each node's row includes the spinloop version string from its daemon + +#### Scenario: Version is omitted for unreachable nodes + +- **WHEN** a node's daemon is unreachable +- **THEN** that node's row shows its failure outcome without a version + +#### Scenario: The fleet-scoped spelling names its replacement + +- **WHEN** the operator runs `spinloop fleet status` +- **THEN** it fails naming `spinloop status` as the command that replaced it + +### Requirement: Fleet dashboard + +`spinloop dashboard` SHALL open an interactive, full-screen view of the resolved target: +one panel per node, arranged in a grid and refreshed +continuously without operator input. It is the one place where what the fleet is +doing and acting on it meet: unlike `fleet metrics --watch`, it takes keyboard +input and drives the node the operator has selected. + +The target SHALL resolve as it does for every other command that acts on a +fleet, so a single named environment opens as a board of one panel. The +dashboard SHALL be openable and usable from cold — a fleet where nothing is up: +every panel SHALL show its node's outcome and reason rather than metrics, and a +node SHALL be startable from within the dashboard. + +A problem with the target itself (a missing or unparseable fleet file, an +environment that is not registered) SHALL fail the command before the view +opens, as it does for the other commands that act on a fleet; a problem +with any node SHALL NOT. + +#### Scenario: A mixed fleet renders every node + +- **WHEN** the operator opens the dashboard against a fleet with some nodes + answering and some not +- **THEN** every node has a panel: answering nodes show their metrics, the others + show their outcome and reason, and the view keeps running + +#### Scenario: A named environment opens as one panel + +- **WHEN** the operator runs `spinloop dashboard --env prod` in a directory + holding no fleet file +- **THEN** the view opens on that environment alone, and the absent file is not + an error + +#### Scenario: Opening on a fleet where nothing is up + +- **WHEN** the operator opens the dashboard and no node is reachable +- **THEN** every panel shows its node's outcome and reason +- **AND** the dashboard does not exit, and starting a node from it works + +#### Scenario: A fleet-file problem fails before the view + +- **WHEN** the named fleet file is missing or unparseable +- **THEN** the command fails naming the problem, and no interactive view opens + +#### Scenario: A non-interactive context is refused + +- **WHEN** the dashboard is run with its input or output not on a terminal, such + as through a pipe or in the background +- **THEN** it fails with a message pointing at `fleet metrics --watch`, and it + does not enter raw terminal mode or emit screen escapes + +#### Scenario: The fleet-scoped spelling names its replacement + +- **WHEN** the operator runs `spinloop fleet dashboard` +- **THEN** it fails naming `spinloop dashboard` as the command that replaced it diff --git a/openspec/changes/top-level-status-and-dashboard/specs/fleet-config/spec.md b/openspec/changes/top-level-status-and-dashboard/specs/fleet-config/spec.md new file mode 100644 index 00000000..ce2eacf1 --- /dev/null +++ b/openspec/changes/top-level-status-and-dashboard/specs/fleet-config/spec.md @@ -0,0 +1,115 @@ +## MODIFIED Requirements + +### Requirement: Fleet file resolution + +Every command that acts on a fleet — the `spinloop fleet` group, and the +top-level verbs that read one — SHALL resolve the fleet the same way, from the +flags given: `--env ` names a single registered cloud environment, +`--fleet ` names a fleet file, and with neither the fleet file is +`./fleet.yaml` in the working directory. One rule serves every such command, so +a target means the same thing wherever it is given. + +Where no target resolves — no `--env`, no `--fleet`, and no fleet file at the +expected path — the command SHALL fail naming every way to give one: the +expected path, `--fleet `, and `--env `. It SHALL NOT look for an +engine on the machine it is running on. A local engine is already shown by the +command that runs it, and a machine whose engines are to be read this way names +them in a fleet file like any other. + +`--env` and `--fleet` SHALL NOT both be given: each names where the model is +served from, so a command stating both SHALL fail naming both rather than +resolving one by precedence. The same rule governs a launch, so an operator +meets it once. + +`--env` SHALL name a registered environment, which is a plain identifier and +never a path. A value that is not a plain identifier, and a name with no +registered configuration, SHALL each fail saying so and how to create the +environment, before any node is contacted. + +`--fleet` SHALL carry a `-f` short form on every `spinloop fleet` subcommand +except `fleet logs`, where `-f` is the short form of `--follow`; on `fleet +logs` the fleet file SHALL be named by the long form `--fleet` only. `--env` +SHALL carry no short form on the fleet commands, so that one spelling means +one thing across the group. + +#### Scenario: Default resolution + +- **WHEN** a `spinloop fleet` command runs in a directory containing + `fleet.yaml` with no `--fleet` or `--env` flag +- **THEN** that file is used + +#### Scenario: Explicit path + +- **WHEN** `spinloop fleet status --fleet ./cluster.yaml` runs +- **THEN** that file is used + +#### Scenario: Short form + +- **WHEN** `spinloop fleet status -f ./cluster.yaml` runs +- **THEN** `./cluster.yaml` is used, exactly as with `--fleet` + +#### Scenario: logs keeps -f for follow + +- **WHEN** the operator runs `spinloop fleet logs -f` +- **THEN** that is the command's follow flag, not a fleet-file flag, and + `logs` takes its fleet file only as `--fleet` + +#### Scenario: Missing file + +- **WHEN** a `spinloop fleet` command runs with no fleet file at the resolved + path +- **THEN** it fails, naming the expected path + +#### Scenario: A named environment is the target + +- **WHEN** `spinloop fleet status --env prod` runs +- **THEN** the command acts on the registered environment `prod` alone, + whether or not a `fleet.yaml` is present + +#### Scenario: A named environment needs no fleet file + +- **WHEN** `spinloop fleet dashboard --env prod` runs in a directory holding + no `fleet.yaml` +- **THEN** the view opens on that one environment, and the absent file is not + an error + +#### Scenario: Naming both a fleet and an environment fails + +- **WHEN** `spinloop fleet status --env prod --fleet ./cluster.yaml` runs +- **THEN** it fails naming both the environment and the fleet file, and + contacts nothing + +#### Scenario: The directory's fleet file does not conflict with --env + +- **WHEN** `spinloop fleet status --env prod` runs in a directory containing + `fleet.yaml` +- **THEN** the environment is the target and the directory's file is ignored, + because only a flag states a conflict + +#### Scenario: An unregistered environment is reported + +- **WHEN** `spinloop fleet status --env nope` runs and no configuration is + registered under that name +- **THEN** it fails saying the environment is not registered and how to + create it, and contacts nothing + +#### Scenario: A path is not an environment name + +- **WHEN** `spinloop fleet status --env ./remote.json` runs +- **THEN** it fails saying an environment name is a plain identifier with no + path + +#### Scenario: No target at all names every way to give one + +- **WHEN** a command that acts on a fleet runs with no `--env`, no `--fleet`, + and no fleet file at the expected path +- **THEN** it fails naming the expected path, `--fleet ` and + `--env `, and contacts nothing + +#### Scenario: No engine on this machine is looked for + +- **WHEN** a read verb runs with no target resolvable on a machine that is + running an engine with its control API up +- **THEN** it fails as above rather than reporting that engine: a local engine + is read through the command that runs it, or by naming the machine in a fleet + file diff --git a/openspec/changes/top-level-status-and-dashboard/specs/remote-endpoint/spec.md b/openspec/changes/top-level-status-and-dashboard/specs/remote-endpoint/spec.md new file mode 100644 index 00000000..52f9aa2b --- /dev/null +++ b/openspec/changes/top-level-status-and-dashboard/specs/remote-endpoint/spec.md @@ -0,0 +1,215 @@ +## MODIFIED Requirements + +### Requirement: Remote command group + +The system SHALL provide a `remote` command group with the subcommands +`bootstrap`, `bake`, `auth`, `start`, `stop`, `restart`, `deploy`, +`ls`, `metrics`, and `keep`. `start`, `stop`, `restart`, `metrics` and +`deploy` each take an optional Spinloop path: +`start` SHALL boot the endpoint and block until it is serving, then perform a +quick TCP probe of the inference endpoint — if the probe fails, a warning is +printed to stderr explaining the network mismatch (see the Remote Start Probe +specification) — and finally print the base URL and API key as shell exports; +`start` SHALL also accept a `--keep DURATION` flag that sets the instance +retention deadline to `now + DURATION`, preventing the idle sweep from +terminating it before that time (see the Remote Keep specification); +`stop` SHALL stop it immediately rather than waiting for its idle timer; +`restart` SHALL stop the endpoint in the manner of a pause — without +terminating it, so its boot disk, its weights and its stable address are +preserved — and SHALL immediately start it again, blocking until it is serving +and reporting progress as `start` does (see the Reporting a start in progress +specification); `restart` SHALL accept a `--force` flag with a `-F` short form +that, when set, performs the stop without first asking the engine to shut down +(see the Endpoint Lifecycle specification for forced stops); +`keep` SHALL set the `Retain-Until` tag on the environment's instance for the +given duration, without starting or stopping the instance (see the Remote Keep +specification); `metrics` SHALL report instance state, token usage, resource +consumption, and GPU information for a running instance; `deploy` SHALL set +what the endpoint serves. `ls` SHALL list the registered remote environments +(see the Remote Environments specification). `bootstrap` SHALL stand up the +account-level AWS control plane (once per account) by obtaining and driving the +CDK project, and takes its own flags rather than a Spinloop path (see the +Endpoint Provisioning specification). `bake` SHALL start an AMI bake for each +runner named, and takes runner names rather than a Spinloop path (see the +Endpoint Provisioning specification). `auth` SHALL store, report, and clear the +long-lived control-plane credential, and takes its own flags rather than a +Spinloop path (see the Remote Auth specification). An unrecognised subcommand +SHALL fail naming the accepted ones. + +#### Scenario: Starting the endpoint + +- **WHEN** the user runs `spinloop remote start` and the endpoint reports ready +- **THEN** the base URL and API key are printed as `export` lines + +#### Scenario: Starting warns when the network is not admitted + +- **WHEN** the user runs `spinloop remote start` and the endpoint reports ready + but the TCP probe to the inference port fails +- **THEN** a warning is printed to stderr with a remediation command, and the + command still exits 0 + +#### Scenario: Starting with a keep flag + +- **WHEN** the user runs `spinloop remote start --keep 4h` and the endpoint reports ready +- **THEN** the base URL and API key are printed as `export` lines, and the + instance retention deadline is set to 4 hours from now + +#### Scenario: Waiting through a cold start + +- **WHEN** the endpoint reports that it is still starting +- **THEN** the command waits and retries until it is ready or the timeout + passes, rather than failing on the first attempt + +#### Scenario: Restarting the endpoint + +- **WHEN** the user runs `spinloop remote restart` for a running environment and + the endpoint reports ready again +- **THEN** the instance was stopped and re-woken without being terminated, the + command blocked until the model was serving again, and the environment's + address is the one its configuration records + +#### Scenario: Forcing a restart skips the engine stop + +- **WHEN** the user runs `spinloop remote restart --force` (or `-F`) +- **THEN** the instance is stopped without the engine being asked to shut down + first, and the command then blocks until the model is serving again + +#### Scenario: Restarting a stopped endpoint starts it + +- **WHEN** the user runs `spinloop remote restart` for an environment whose instance is already stopped +- **THEN** the instance is re-woken rather than replaced, and the command blocks + until the model is serving again, as with a plain start + +#### Scenario: A failed re-wake says how to recover + +- **WHEN** the stop half of a restart has taken effect but the wake fails +- **THEN** the command fails saying the instance is stopped and that + `spinloop remote start` will bring it back + +#### Scenario: Listing environments + +- **WHEN** the user runs `spinloop remote ls` +- **THEN** the registered environments are listed rather than any endpoint being + contacted + +#### Scenario: Setting a keep deadline + +- **WHEN** the user runs `spinloop remote keep 2h` +- **THEN** the instance retention tag is set and the deadline is reported + +#### Scenario: Metrics reports instance figures + +- **WHEN** the user runs `spinloop remote metrics` with a running instance +- **THEN** token counts, resource usage, and GPU information are displayed + +#### Scenario: Bootstrap is a recognised subcommand + +- **WHEN** the user runs `spinloop remote bootstrap` +- **THEN** the command is dispatched to the provisioning flow rather than + reported as unknown + +#### Scenario: Bake is a recognised subcommand + +- **WHEN** the user runs `spinloop remote bake llamacpp` +- **THEN** the command is dispatched to the bake flow rather than + reported as unknown + +#### Scenario: Auth is a recognised subcommand + +- **WHEN** the user runs `spinloop remote auth` +- **THEN** the command is dispatched to the credential store, report, and clear + flow rather than reported as unknown + +#### Scenario: Unknown subcommand + +- **WHEN** the user runs `spinloop remote frobnicate` +- **THEN** the command fails listing the accepted subcommands, which include + `bootstrap`, `bake`, `metrics`, and `keep` + +## REMOVED Requirements + +### Requirement: Status reports when the endpoint last did work + +**Reason**: `spinloop remote status` is removed — one command reads an +endpoint's state, whether it is named as an environment or as a fleet node. +The behaviour is unchanged and is restated below under a name that does not +carry the removed command's spelling. + +**Migration**: `spinloop status --env `. + +### Requirement: Status degrades when activity cannot be read + +**Reason**: As above — the requirement describes `spinloop remote status`, +which no longer exists. Restated below, unchanged in substance. + +**Migration**: `spinloop status --env `. + +## ADDED Requirements + +### Requirement: An environment reports when it last did work + +`spinloop status --env ` SHALL report how long it has been since the endpoint's +engine last did any work, alongside the instance state and health it reports +already. The figure SHALL come from the activity the on-instance daemon +tracks, not from a measurement the control plane makes itself — one answer, +derived on the box, however it is asked for. + +The figure SHALL be labelled "last active", matching the wording and duration +formatting used everywhere else this fact appears, so the same fact reads the +same way in every command. + +Collecting it SHALL NOT make `status` slower than its health check already +makes it: the daemon SHALL be asked in parallel with the health check rather +than after it. Nor SHALL it introduce a side effect — `status` SHALL remain a +read, and SHALL still perform no TCP probe. + +#### Scenario: A running endpoint reports its last activity + +- **WHEN** the user runs `spinloop status --env ` against a running endpoint + whose engine has served work +- **THEN** the output reports how long ago that work happened, labelled "last + active", beside the state and health lines + +#### Scenario: Status stays a read + +- **WHEN** the user runs `spinloop status --env ` +- **THEN** nothing is started, stopped or probed in order to obtain the + last-active figure + +### Requirement: An environment's status degrades when activity cannot be read + +`spinloop status --env ` SHALL omit the last-active figure rather than fail, +report zero, or imply inactivity, whenever the figure cannot be obtained. That +covers an endpoint whose engine has not yet done any work, a daemon that +cannot be reached or answers unrecognisably, and an instance that is not +running — reaching the daemon needs a running box, so a stopped or undeployed +environment has nothing to report about its engine. + +A failure to read the activity SHALL NOT affect the rest of the report: the +state and health lines SHALL be exactly what they are today, and the command +SHALL still succeed. + +#### Scenario: A stopped instance reports no activity figure + +- **WHEN** the user runs `spinloop status --env ` and the instance is stopped or + undeployed +- **THEN** the output reports the state as it does today and shows no + last-active figure + +#### Scenario: An unreachable daemon does not spoil the report + +- **WHEN** the endpoint is running but its daemon cannot be reached +- **THEN** the state and health lines are reported as they are today, no + last-active figure is shown, and the command succeeds + +#### Scenario: An engine that has done nothing yet + +- **WHEN** the endpoint is running and its daemon reports no last-active time +- **THEN** no last-active figure is shown, rather than one implying the engine + has been quiet since it started + +#### Scenario: The remote spelling names its replacement + +- **WHEN** the operator runs `spinloop remote status` +- **THEN** it fails naming `spinloop status --env ` as the command that + replaced it diff --git a/openspec/changes/top-level-status-and-dashboard/specs/remote-version-reporting/spec.md b/openspec/changes/top-level-status-and-dashboard/specs/remote-version-reporting/spec.md new file mode 100644 index 00000000..d575784d --- /dev/null +++ b/openspec/changes/top-level-status-and-dashboard/specs/remote-version-reporting/spec.md @@ -0,0 +1,52 @@ +## REMOVED Requirements + +### Requirement: Remote status shows version + +**Reason**: `spinloop remote status` is removed; the version it displayed is +displayed by the command that replaced it. Restated below under a name that +does not carry the removed spelling. + +**Migration**: `spinloop status --env `. + +### Requirement: Fleet status shows version + +**Reason**: `spinloop fleet status` is removed in favour of the top-level +`spinloop status`, which serves every target kind. The per-node version +behaviour is unchanged and is restated below. + +**Migration**: `spinloop status`. + +## ADDED Requirements + +### Requirement: An environment's status shows version + +`spinloop status --env ` SHALL display the spinloop version running on the remote instance alongside its existing state, health, and base URL fields. + +#### Scenario: Version is shown when the instance is running + +- **WHEN** the user runs `spinloop status --env ` against a running instance +- **THEN** the output includes a `version` line with the spinloop version string (e.g. `version: 1.16.0`) + +#### Scenario: Version is unavailable when the instance is stopped + +- **WHEN** the user runs `spinloop status --env ` against a stopped instance +- **THEN** the output omits the version line, since the daemon is not reachable + +### Requirement: Status shows version per node + +`spinloop status` SHALL display the spinloop version for each node alongside its existing state and serving columns, read from the daemon's `/v1/status` response. + +#### Scenario: Version is shown per node + +- **WHEN** `spinloop status` runs against a fleet of running nodes +- **THEN** each node's row includes the spinloop version string + +#### Scenario: Version is omitted for unreachable nodes + +- **WHEN** a node's daemon is unreachable +- **THEN** that node's row shows its failure outcome without a version + +#### Scenario: Versions differ across nodes + +- **WHEN** nodes in the fleet run different spinloop versions +- **THEN** each node's row shows its own version, making the difference visible diff --git a/openspec/changes/top-level-status-and-dashboard/tasks.md b/openspec/changes/top-level-status-and-dashboard/tasks.md new file mode 100644 index 00000000..e454ccc5 --- /dev/null +++ b/openspec/changes/top-level-status-and-dashboard/tasks.md @@ -0,0 +1,65 @@ +## 1. Carrying the cloud-only facts + +- [ ] 1.1 Add the endpoint's health and base URL to `fleet.NodeResult`, filled + by `statusFromRemote` from the reply it already receives and left empty + by a daemon node (design D2). Verify a unit test over both node kinds: + the cloud node carries both, the daemon node neither. +- [ ] 1.2 Render them in the status table when present, omitted when not. + Verify the same environment read through a fleet file shows what + `remote status` shows today for health and address. + +## 2. The verbs + +- [ ] 2.1 Add `cmd/spinloop/status.go`: a root-registered `status` taking + `--env`/`--fleet`, resolving through `resolveFleetTarget` and rendering + through `renderFleetStatus`. Verify `spinloop status --env ` and + `spinloop status --fleet ` both render, and that naming both fails + with the shared message. +- [ ] 2.2 Add `cmd/spinloop/dashboard.go` the same way, over `dashModelFor` + and `runDashProgram`. Verify `spinloop dashboard --env ` opens on + one panel in a directory with no fleet file, and that a non-terminal + invocation is refused as before. +- [ ] 2.3 Register both at the root and carry over the completion the fleet + spellings had (`--fleet` from files, `--env` from registered + environments). Verify the completion test covers both new commands. + +## 3. Removing the old spellings + +- [ ] 3.1 Delete `fleetStatusCmd` and `fleetDashboardCmd` and unregister them, + keeping `renderFleetStatus`, `dashModelFor` and `runDashProgram` where + they are. Verify `spinloop fleet --help` lists neither. +- [ ] 3.2 Delete `remoteStatusCmd` and `runRemoteStatus` and unregister them. + Verify `spinloop remote --help` no longer lists `status`, and that + nothing else referenced the removed functions. +- [ ] 3.3 Signpost all three moved spellings (design D4): `fleet status` and + `fleet dashboard` through the existing `movedSubcommands`, and + `remote status` by giving the `remote` group the same `Args: groupArgs` + the fleet group has. Verify each names its replacement, and that an + unknown subcommand in either group still gets cobra's own error. + +## 4. The no-target message + +- [ ] 4.1 Change `fleet.Resolve`'s failure to name the expected path, + `--fleet ` and `--env ` (design D5). Verify a test asserts + all three appear, and that the existing tests asserting on that message + still pass or are updated to the fuller wording. +- [ ] 4.2 Verify no implicit local target: a read verb with nothing resolvable + fails as above even on a machine whose daemon is answering on its + default port. + +## 5. Consumers, docs and verification + +- [ ] 5.1 Update every invocation of the three moved spellings across `docs/`, + `README.md` and `examples/` — including the CI-run + `examples/fleet-docker/run-tests.sh` and `examples/gateway-docker/run-tests.sh` + if they use them. Verify no consumer still invokes a removed spelling. +- [ ] 5.2 Add `docs/commands/status.md` and `docs/commands/dashboard.md`, and + point `fleet.md` and `remote.md` at them. Verify `docs/README.md`'s + command table lists both. +- [ ] 5.3 Run `gofmt -l .` (expect no output), `go vet ./...` and + `go test ./... -cover`, confirming total coverage is unchanged and still + >= 80%. +- [ ] 5.4 Verify `metrics` and `logs` are untouched: no file under + `cmd/spinloop/` that implements them changed except where a shared + renderer moved, and `fleet metrics`/`fleet logs`/`remote metrics`/`remote + logs` all still work. From 25abcd548bc4a4832d5abe97d8bf97f722d23944 Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Tue, 15 Sep 2026 01:04:52 +0100 Subject: [PATCH 2/3] feat: make status and dashboard top-level verbs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reading an engine's state is one operation the CLI spelled twice. fleet status and remote status answer the same question about the same kind of thing, and since --env landed both worked on the same environment and printed differently. The renderers converged long ago — status_render.go exists because someone noticed — but the commands above them did not. status and dashboard now take their target the way every command that acts on a fleet does: --env, --fleet, or the working directory's fleet.yaml. A single environment renders as a one-row table and opens as a board of one panel, with no fleet file needed. fleet status and fleet dashboard are removed, each naming its replacement. With no target resolvable the verbs fail rather than looking for an engine on this machine, and the failure now names all three ways to give one — it offered only --fleet and creating a file, though --env has been a target since it landed. remote status stays. It makes a second control call for the version, applies a Spinloop's ENV before resolving, and renders the endpoint's address and keep deadline: three things a fan-out does not do. remote metrics does the same three, so the design for carrying them belongs with the change that moves metrics and logs — once, for all of them. --- README.md | 12 +- cmd/spinloop/commands.go | 9 +- cmd/spinloop/dashboard.go | 55 +++++ cmd/spinloop/fleet.go | 30 --- cmd/spinloop/fleet_dashboard.go | 42 ---- cmd/spinloop/fleet_dashboard_test.go | 2 +- cmd/spinloop/fleet_test.go | 18 +- cmd/spinloop/status.go | 58 +++++ cmd/spinloop/target_test.go | 14 +- docs/README.md | 4 +- docs/commands/dashboard.md | 52 +++++ docs/commands/fleet.md | 41 ++-- docs/commands/serve.md | 4 +- docs/commands/status.md | 64 ++++++ examples/fleet-docker/README.md | 8 +- examples/fleet-docker/compose.yaml | 2 +- examples/fleet-docker/run-tests.sh | 20 +- examples/fleet-local/README.md | 2 +- examples/fleet-mixed/README.md | 2 +- examples/fleet-mixed/fleet.yaml | 2 +- examples/fleet-remote/README.md | 2 +- examples/fleet-remote/fleet.yaml | 2 +- examples/fleet/README.md | 2 +- examples/fleet/fleet.yaml | 2 +- examples/gateway-docker/README.md | 2 +- examples/gateway-docker/run-tests.sh | 8 +- internal/fleet/config.go | 15 +- internal/fleet/for_environment_test.go | 10 +- .../top-level-status-and-dashboard/design.md | 71 +++--- .../proposal.md | 29 ++- .../specs/fleet-client/spec.md | 15 +- .../specs/remote-endpoint/spec.md | 215 ------------------ .../specs/remote-version-reporting/spec.md | 52 ----- .../top-level-status-and-dashboard/tasks.md | 51 ++--- 34 files changed, 412 insertions(+), 505 deletions(-) create mode 100644 cmd/spinloop/dashboard.go create mode 100644 cmd/spinloop/status.go create mode 100644 docs/commands/dashboard.md create mode 100644 docs/commands/status.md delete mode 100644 openspec/changes/top-level-status-and-dashboard/specs/remote-endpoint/spec.md delete mode 100644 openspec/changes/top-level-status-and-dashboard/specs/remote-version-reporting/spec.md diff --git a/README.md b/README.md index 83c9761c..f8466ff9 100644 --- a/README.md +++ b/README.md @@ -63,14 +63,14 @@ want one — starting when you ask, and still up for your next session. spinloop daemon # ready on :4242; no model runs until you ask # on the machine you work at — fleet.yaml names the nodes -spinloop fleet status # one row per node: state, and what it serves -spinloop fleet dashboard # the same fleet, as a board you leave open +spinloop status # one row per node: state, and what it serves +spinloop dashboard # the same fleet, as a board you leave open ``` Start the whole fleet with `spinloop up` or just one with `spinloop fleet start gpu-box` — this gets you the model from step 1 running on a machine across the room, or over Tailscale.

- The spinloop fleet dashboard: four nodes serving Qwen3.8-27B under llama.cpp, and a fifth cloud node not yet deployed + The spinloop dashboard: four nodes serving Qwen3.8-27B under llama.cpp, and a fifth cloud node not yet deployed

> In this screenshot, five nodes are configured and four are up, each serving the same @@ -537,8 +537,8 @@ that node's model needs a key of its own, name it with `engineTokenEnv`: driving a machine and talking to the model on it are separate credentials. ```sh -spinloop fleet status # one row per node: state and what it serves -spinloop fleet dashboard # the interactive tiled view — watch it, drive it +spinloop status # one row per node: state and what it serves +spinloop dashboard # the interactive tiled view — watch it, drive it spinloop fleet start gpu-box # start one node's engine ``` @@ -596,7 +596,7 @@ engine — in about a minute: cd examples/fleet-docker && cp .env.example .env docker compose up -d --build set -a && . ./.env && set +a -spinloop fleet status --fleet ./fleet.yaml +spinloop status --fleet ./fleet.yaml ``` Only one machine? A fleet of one is still worth it — diff --git a/cmd/spinloop/commands.go b/cmd/spinloop/commands.go index 7cc694ce..2475f822 100644 --- a/cmd/spinloop/commands.go +++ b/cmd/spinloop/commands.go @@ -74,6 +74,8 @@ has been.`, unaliasCmd(), serveCmd(), upCmd(), + statusCmd(), + dashboardCmd(), codeCmd(), daemonCmd(), gatewayCmd(), @@ -344,7 +346,9 @@ func groupArgs(c *cobra.Command, args []string) error { // of movedTopLevelCommands: the same signpost, for a spelling that was two // words rather than one. var movedSubcommands = map[string]string{ - "fleet harness": "code --fleet ", + "fleet harness": "code --fleet ", + "fleet status": "status", + "fleet dashboard": "dashboard", } // fleetCmd builds the fleet parent and its subcommands. The parent does @@ -366,10 +370,8 @@ error — only a problem with the fleet file itself fails a command.`, RunE: groupFallback, } fleet.AddCommand( - fleetStatusCmd(), fleetMetricsCmd(), fleetLogsCmd(), - fleetDashboardCmd(), fleetRouteCmd(), fleetStartCmd(), fleetStopCmd(), @@ -389,6 +391,7 @@ the same Spinloop. The endpoint's URLs come from the Spinloop's REMOTE — a bar name selects an environment under ~/.config/spinloop/remotes//, a path names a file — falling back to the default environment. Each subcommand's --help says what that step does.`, + Args: groupArgs, SilenceErrors: true, SilenceUsage: true, RunE: groupFallback, diff --git a/cmd/spinloop/dashboard.go b/cmd/spinloop/dashboard.go new file mode 100644 index 00000000..9973d3ed --- /dev/null +++ b/cmd/spinloop/dashboard.go @@ -0,0 +1,55 @@ +// `spinloop dashboard`: the interactive board over whatever the target names — +// a fleet, or a single registered environment, which opens as a board of one. +// The model and the renderers live in dashboard_model.go and +// dashboard_render.go, and the program in fleet_dashboard.go; this is the +// command. + +package main + +import ( + "github.com/spf13/cobra" +) + +func dashboardCmd() *cobra.Command { + var path, envName string + c := &cobra.Command{ + Use: "dashboard", + Short: "watch the engines in an interactive tiled view", + Long: `An interactive live view of the target: a tile per node, each drawing +what the gauge format of fleet metrics prints — state, what it serves, the +resource gauges, the token counters — repainted on an interval. A single +registered environment (--env) opens as a board of one. + +The view is read-only apart from four keys: s starts the selected node, k +keeps a remote environment for a duration you type — it asks how long, +pre-filled with 4h, and reports the deadline the control plane set when the +keep is done — a abandons a start still in flight on it (the wait ends, the +node is free again — a wake the cloud is carrying goes on), x stops it after +a confirmation. The arrow keys move the selection, r forces a refresh, q or +Ctrl+C leaves. The keep key shows only for a node that can be kept — a remote +environment — and a kept environment's tile and detail view carry its +deadline beside the last-active line, whatever the engine's state. + +A node that cannot be reached is still a tile, showing why, and a node whose +token reference is unresolvable holds its reason for the life of the view. +The board needs a terminal; to stream the metrics into a pipe, use fleet +metrics --watch instead.`, + Args: cobra.NoArgs, + SilenceErrors: true, + SilenceUsage: true, + RunE: func(c *cobra.Command, _ []string) error { + resolve(c) + return runFleetDashboard(fleetTarget{envName: envName, fleetPath: path}) + }, + } + fs := c.Flags() + fs.StringVarP(&path, "fleet", "f", "", fleetFileUsage) + fs.StringVar(&envName, "env", "", envFlagTargetUsage) + c.ValidArgsFunction = noPositionals + compRegister(c, "fleet", compFiles) + compRegister(c, "env", compEnvs) + return c +} + +// cmdDashboard runs the command through the tree — the seam the suite calls. +func cmdDashboard(args []string) error { return execCmd(dashboardCmd(), args) } diff --git a/cmd/spinloop/fleet.go b/cmd/spinloop/fleet.go index a77ea8a2..7bbecef2 100644 --- a/cmd/spinloop/fleet.go +++ b/cmd/spinloop/fleet.go @@ -37,36 +37,6 @@ func cmdFleet(args []string) error { // fleetFileFlag is the --fleet flag's help, shared by every fleet subcommand. const fleetFileUsage = "path to the fleet file (default ./fleet.yaml)" -// fleetStatusCmd reports every node's engine state, one row per node. A node -// that cannot be reached is a row, not a failure: the rest of the fleet still -// renders and the command still succeeds. -func fleetStatusCmd() *cobra.Command { - var path, envName string - c := &cobra.Command{ - Use: "status", - Short: "report every node's engine state", - Args: cobra.ArbitraryArgs, - SilenceErrors: true, - SilenceUsage: true, - RunE: func(c *cobra.Command, _ []string) error { - resolve(c) - cfg, err := resolveFleetTarget(fleetTarget{envName: envName, fleetPath: path}) - if err != nil { - return err - } - results := cfg.FanOut(context.Background(), fleet.StatusCall) - renderFleetStatus(os.Stdout, results) - return nil - }, - } - c.Flags().StringVarP(&path, "fleet", "f", "", fleetFileUsage) - c.Flags().StringVar(&envName, "env", "", envFlagTargetUsage) - c.ValidArgsFunction = noPositionals - compRegister(c, "fleet", compFiles) - compRegister(c, "env", compEnvs) - return c -} - // renderFleetStatus writes the status table: node, state, what it serves, and // the reason when a node did not answer. func renderFleetStatus(w io.Writer, results []fleet.NodeResult) { diff --git a/cmd/spinloop/fleet_dashboard.go b/cmd/spinloop/fleet_dashboard.go index e384ca6d..51c59999 100644 --- a/cmd/spinloop/fleet_dashboard.go +++ b/cmd/spinloop/fleet_dashboard.go @@ -11,52 +11,10 @@ import ( "os" tea "github.com/charmbracelet/bubbletea" - "github.com/spf13/cobra" "github.com/spinloop-ai/spinloop/internal/fleet" "golang.org/x/term" ) -// fleetDashboardCmd builds the `fleet dashboard` subcommand. -func fleetDashboardCmd() *cobra.Command { - var path, envName string - c := &cobra.Command{ - Use: "dashboard", - Short: "watch the fleet in an interactive tiled view", - Long: `An interactive live view of the fleet: a tile per node, each drawing -what the gauge format of fleet metrics prints — state, what it serves, the -resource gauges, the token counters — repainted on an interval. - -The view is read-only apart from four keys: s starts the selected node, k -keeps a remote environment for a duration you type — it asks how long, -pre-filled with 4h, and reports the deadline the control plane set when the -keep is done — a abandons a start still in flight on it (the wait ends, the -node is free again — a wake the cloud is carrying goes on), x stops it after -a confirmation. The arrow keys move the selection, r forces a refresh, q or -Ctrl+C leaves. The keep key shows only for a node that can be kept — a remote -environment — and a kept environment's tile and detail view carry its -deadline beside the last-active line, whatever the engine's state. - -A node that cannot be reached is still a tile, showing why, and a node whose -token reference is unresolvable holds its reason for the life of the view. -The board needs a terminal; to stream the metrics into a pipe, use fleet -metrics --watch instead.`, - Args: cobra.ArbitraryArgs, - SilenceErrors: true, - SilenceUsage: true, - RunE: func(c *cobra.Command, _ []string) error { - resolve(c) - return runFleetDashboard(fleetTarget{envName: envName, fleetPath: path}) - }, - } - fs := c.Flags() - fs.StringVarP(&path, "fleet", "f", "", fleetFileUsage) - fs.StringVar(&envName, "env", "", envFlagTargetUsage) - c.ValidArgsFunction = noPositionals - compRegister(c, "fleet", compFiles) - compRegister(c, "env", compEnvs) - return c -} - // runFleetDashboard opens the view. The terminal check comes first — before // the fleet file is even read — so a piped invocation fails the same way // wherever the fleet file stands: it never half-enters the view. diff --git a/cmd/spinloop/fleet_dashboard_test.go b/cmd/spinloop/fleet_dashboard_test.go index ae5ceb5b..b248df05 100644 --- a/cmd/spinloop/fleet_dashboard_test.go +++ b/cmd/spinloop/fleet_dashboard_test.go @@ -2205,7 +2205,7 @@ func TestDashProgramStartsAndStopsANode(t *testing.T) { // the command that carries the same data into a pipe. func TestFleetDashboardRefusesPipedOutput(t *testing.T) { captureStdout(t, func() { - err := cmdFleet([]string{"dashboard"}) + err := cmdDashboard(nil) if err == nil { t.Fatal("dashboard ran without a terminal") } diff --git a/cmd/spinloop/fleet_test.go b/cmd/spinloop/fleet_test.go index 29d9f18d..31557f64 100644 --- a/cmd/spinloop/fleet_test.go +++ b/cmd/spinloop/fleet_test.go @@ -106,7 +106,7 @@ func twoNodeFleet(t *testing.T, state string) *httptest.Server { func TestCmdFleetStatusRendersEveryNode(t *testing.T) { twoNodeFleet(t, "running") out := captureStdout(t, func() { - if err := cmdFleet([]string{"status"}); err != nil { + if err := cmdStatus(nil); err != nil { // One unreachable node must not fail the command. t.Errorf("fleet status returned %v", err) } @@ -133,7 +133,7 @@ func TestCmdFleetStatusShowsVersion(t *testing.T) { "nodes:\n - name: node\n host: %s\n port: %d\n", host, port)) out := captureStdout(t, func() { - if err := cmdFleet([]string{"status"}); err != nil { + if err := cmdStatus(nil); err != nil { t.Error(err) } }) @@ -628,7 +628,7 @@ func TestCmdFleetUnknownNodeNamesTheKnownOnes(t *testing.T) { func TestCmdFleetMissingFileNamesThePath(t *testing.T) { t.Chdir(t.TempDir()) - err := cmdFleet([]string{"status"}) + err := cmdStatus(nil) if err == nil { t.Fatal("missing fleet file accepted") } @@ -649,7 +649,7 @@ func TestCmdFleetExplicitPath(t *testing.T) { // Somewhere else entirely, so only --fleet can find it. t.Chdir(t.TempDir()) out := captureStdout(t, func() { - if err := cmdFleet([]string{"status", "--fleet", path}); err != nil { + if err := cmdStatus([]string{"--fleet", path}); err != nil { t.Error(err) } }) @@ -672,7 +672,7 @@ func TestCmdFleetExplicitPathShortForm(t *testing.T) { // Somewhere else entirely, so only -f can find it. t.Chdir(t.TempDir()) out := captureStdout(t, func() { - if err := cmdFleet([]string{"status", "-f", path}); err != nil { + if err := cmdStatus([]string{"-f", path}); err != nil { t.Error(err) } }) @@ -687,7 +687,9 @@ func TestFleetFlagShortForm(t *testing.T) { isolateConfig(t) root := newRootCmd() fleet := commandUnder(t, root, "fleet") - for _, name := range []string{"status", "metrics", "start", "stop", "deploy", "route", "dashboard"} { + // status and dashboard are top-level verbs now; the rest still hang off + // the group, and every one of them offers -f for the fleet file. + for _, name := range []string{"metrics", "start", "stop", "deploy", "route"} { sub := commandUnder(t, fleet, name) f := sub.Flags().Lookup("fleet") if f == nil { @@ -786,7 +788,7 @@ func TestCmdFleetStatusShowsIdleTime(t *testing.T) { writeFleetFile(t, fmt.Sprintf("nodes:\n - name: busy\n host: %s\n port: %d\n", host, port)) out := captureStdout(t, func() { - if err := cmdFleet([]string{"status"}); err != nil { + if err := cmdStatus(nil); err != nil { t.Error(err) } }) @@ -810,7 +812,7 @@ func TestCmdFleetStatusOmitsIdleWithoutActivity(t *testing.T) { writeFleetFile(t, fmt.Sprintf("nodes:\n - name: fresh\n host: %s\n port: %d\n", host, port)) out := captureStdout(t, func() { - if err := cmdFleet([]string{"status"}); err != nil { + if err := cmdStatus(nil); err != nil { t.Error(err) } }) diff --git a/cmd/spinloop/status.go b/cmd/spinloop/status.go new file mode 100644 index 00000000..3ffd6a96 --- /dev/null +++ b/cmd/spinloop/status.go @@ -0,0 +1,58 @@ +// `spinloop status`: what every engine in the target is doing, one row each. +// The target is whatever names one — a registered environment, a fleet file, +// or the fleet file in the working directory — so an environment and a fleet +// holding that same environment read identically. The rendering lives beside +// the fleet's other views in fleet.go; this is the command. + +package main + +import ( + "context" + "os" + + "github.com/spf13/cobra" + "github.com/spinloop-ai/spinloop/internal/fleet" +) + +func statusCmd() *cobra.Command { + var path, envName string + c := &cobra.Command{ + Use: "status", + Short: "report every engine's state", + Long: `reports what each engine in the target is doing: its state, what it +serves, how long since it last did work, and the spinloop version of the +daemon running it. One row per node, queried concurrently, so the command +takes as long as the slowest node rather than all of them added up. + +The target is a registered environment (--env), a fleet file (--fleet), or +the fleet.yaml in the working directory. A node that cannot be reached is a +row saying so, not a failure: one unreachable machine never blanks the rest. + +An environment's endpoint address is spinloop remote env, and its retention +deadline spinloop fleet metrics — neither is a column here, because neither +applies to every node.`, + Args: cobra.NoArgs, + SilenceErrors: true, + SilenceUsage: true, + RunE: func(c *cobra.Command, _ []string) error { + resolve(c) + cfg, err := resolveFleetTarget(fleetTarget{envName: envName, fleetPath: path}) + if err != nil { + return err + } + results := cfg.FanOut(context.Background(), fleet.StatusCall) + renderFleetStatus(os.Stdout, results) + return nil + }, + } + fs := c.Flags() + fs.StringVarP(&path, "fleet", "f", "", fleetFileUsage) + fs.StringVar(&envName, "env", "", envFlagTargetUsage) + c.ValidArgsFunction = noPositionals + compRegister(c, "fleet", compFiles) + compRegister(c, "env", compEnvs) + return c +} + +// cmdStatus runs the command through the tree — the seam the suite calls. +func cmdStatus(args []string) error { return execCmd(statusCmd(), args) } diff --git a/cmd/spinloop/target_test.go b/cmd/spinloop/target_test.go index dbc56315..c948bc99 100644 --- a/cmd/spinloop/target_test.go +++ b/cmd/spinloop/target_test.go @@ -127,8 +127,8 @@ func TestResolveFleetTargetUnregisteredEnv(t *testing.T) { if err == nil { t.Fatal("want an error") } - if !strings.Contains(err.Error(), "is not registered") { - t.Errorf("error %q does not say the environment is unregistered", err) + if !strings.Contains(err.Error(), "remotes/nope/remote.json") { + t.Errorf("error %q does not name the environment's registry path", err) } } @@ -197,7 +197,7 @@ func TestFleetCommandsCompleteEnv(t *testing.T) { }) t.Chdir(t.TempDir()) - for _, sub := range []string{"status", "metrics", "logs", "dashboard", "start", "stop", "deploy", "route"} { + for _, sub := range []string{"metrics", "logs", "start", "stop", "deploy", "route"} { t.Run(sub, func(t *testing.T) { got, _ := complete(t, "fleet", sub, "--env", "") want := map[string]bool{"prod": false, "staging": false} @@ -233,7 +233,7 @@ func TestFleetStatusEnvMatchesAOneNodeFile(t *testing.T) { // Named by a fleet file holding exactly that node. writeFleetFile(t, "nodes:\n - name: prod\n kind: remote\n") fromFile := captureStdout(t, func() { - if err := cmdFleet([]string{"status"}); err != nil { + if err := cmdStatus(nil); err != nil { t.Errorf("fleet status returned %v", err) } }) @@ -241,7 +241,7 @@ func TestFleetStatusEnvMatchesAOneNodeFile(t *testing.T) { // Named by the flag, from a directory holding no fleet file at all. t.Chdir(t.TempDir()) fromEnv := captureStdout(t, func() { - if err := cmdFleet([]string{"status", "--env", "prod"}); err != nil { + if err := cmdStatus([]string{"--env", "prod"}); err != nil { t.Errorf("fleet status --env returned %v", err) } }) @@ -270,7 +270,7 @@ func TestFleetStatusEnvNeedsNoFleetFile(t *testing.T) { t.Chdir(t.TempDir()) out := captureStdout(t, func() { - if err := cmdFleet([]string{"status", "--env", "prod"}); err != nil { + if err := cmdStatus([]string{"--env", "prod"}); err != nil { t.Errorf("fleet status --env returned %v", err) } }) @@ -291,7 +291,7 @@ func TestFleetStatusRefusesEnvAndFleet(t *testing.T) { }) writeFleetFile(t, oneNodeFleetBody) - err := cmdFleet([]string{"status", "--env", "prod", "--fleet", "fleet.yaml"}) + err := cmdStatus([]string{"--env", "prod", "--fleet", "fleet.yaml"}) if err == nil { t.Fatal("want an error") } diff --git a/docs/README.md b/docs/README.md index 8cd61ec3..94b49a77 100644 --- a/docs/README.md +++ b/docs/README.md @@ -59,7 +59,9 @@ Four words carry the whole tool: | [`spinloop up`](commands/up.md) | Start the engine this directory holds: the fleet, or the `Spinloop`'s server | | [`spinloop code`](commands/code.md) | Launch the active harness — a one-word shortcut for `spinloop harness open` | | [`spinloop daemon`](commands/serve.md#the-control-api---api-and-spinloop-daemon) | Supervise an engine over the [control API](http-api.md) | -| [`spinloop fleet`](commands/fleet.md) | Observe and drive the engines on every machine you run, and launch your agent against them | +| [`spinloop status`](commands/status.md) | What every engine you run is doing, one row each — a fleet, or one environment | +| [`spinloop dashboard`](commands/dashboard.md) | The live tiled view of the same, with the keys to drive it | +| [`spinloop fleet`](commands/fleet.md) | Drive the engines on every machine you run: start, stop, deploy, route | | [`spinloop gateway`](commands/gateway.md) | Serve the fleet under one OpenAI-compatible endpoint | | [`spinloop orchestrator`](commands/orchestrator.md) | Work a backlog of items against the fleet, at the fleet's declared pace | | [`spinloop work`](commands/work.md) | Work the work items file from the shell: add, list, abort, remove | diff --git a/docs/commands/dashboard.md b/docs/commands/dashboard.md new file mode 100644 index 00000000..110b6a00 --- /dev/null +++ b/docs/commands/dashboard.md @@ -0,0 +1,52 @@ +# spinloop dashboard + +The live, interactive view of what you are running: a tile per node, repainted +on an interval, with the keys to drive the one you have selected. + +```sh +spinloop dashboard # the fleet.yaml in this directory +spinloop dashboard --fleet ./cluster.yaml +spinloop dashboard --env prod # one environment, as a board of one +``` + +Each tile draws what the gauge format of +[`spinloop fleet metrics`](fleet.md#metrics) prints for that node — its state, +what it serves, the resource gauges, the token counters — so the board and the +metrics view cannot word the same reading differently. + +## Keys + +| key | what it does | +| --- | --- | +| `↑ ↓ ← →` | move the selection | +| `enter` | open the selected node's detail view | +| `s` | start the selected node's engine | +| `k` | keep a cloud environment for a duration you type (asks how long, pre-filled `4h`) | +| `a` | abandon a start still in flight — the wait ends, the node is free; a wake the cloud is already carrying goes on | +| `x` | stop the selected node, after a confirmation | +| `r` | force a refresh | +| `q` / `ctrl-c` | leave | + +`k` shows only for a node that can be kept — a cloud environment — and a kept +environment carries its deadline beside the last-active line on both the tile +and the detail view, whatever the engine's state. + +## Which target + +The three ways every command that acts on a fleet takes one — `--env `, +`--fleet `, or the working directory's `fleet.yaml`. A single +environment opens as a board of one panel, which needs no fleet file at all. + +A problem with the target itself — a missing or unparseable fleet file, an +environment that is not registered — fails before the view opens. A problem +with any *node* does not: an unreachable node is a tile showing why, and a +node whose token reference cannot be resolved holds its reason for the life of +the view. + +The board needs a terminal. To stream the same readings into a pipe, use +`spinloop fleet metrics --watch`. + +## See also + +- [`spinloop status`](status.md) — the same facts as one table, scriptable +- [`spinloop fleet`](fleet.md) — the fleet file, and driving nodes from the CLI diff --git a/docs/commands/fleet.md b/docs/commands/fleet.md index 4bd85b9e..6a84a009 100644 --- a/docs/commands/fleet.md +++ b/docs/commands/fleet.md @@ -5,17 +5,22 @@ Observe and drive every engine you run, from one place. Each machine runs `fleet.yaml` names them, and `spinloop fleet` fans out over their control APIs. ```sh -spinloop fleet status # one row per node: state and what it serves -spinloop fleet metrics # each node's engine + system metrics -spinloop fleet metrics -w # the same, redrawn in place until interrupted -spinloop fleet dashboard # the interactive tiled view — watch it, drive it +spinloop status # one row per node: state and what it serves +spinloop dashboard # the interactive tiled view — watch it, drive it +spinloop fleet metrics # each node's engine + system metrics +spinloop fleet metrics -w # the same, redrawn in place until interrupted spinloop fleet route my-spinloop # which node a harness launch would pick -spinloop fleet start gpu-box # start one or more nodes' engines -spinloop fleet start --all # start every node in the fleet -spinloop fleet stop gpu-box # stop one or more nodes' engines -spinloop fleet deploy --all # create every kind: remote node's AWS environment +spinloop fleet start gpu-box # start one or more nodes' engines +spinloop fleet start --all # start every node in the fleet +spinloop fleet stop gpu-box # stop one or more nodes' engines +spinloop fleet deploy --all # create every kind: remote node's AWS environment ``` +[`spinloop status`](status.md) and [`spinloop dashboard`](dashboard.md) are +top-level commands, not part of this group: they read whatever target you name +— a fleet file, or a single registered environment — so there is one command +for "what is running", however it is configured. + A fleet is also where [`spinloop harness open`](harness.md#launching-against-your-fleet) sends an agent: a launch routed through a fleet file picks a node and launches against it, so the machine you are sitting at needs no engine of its own. @@ -41,8 +46,8 @@ A registered environment and a one-node fleet file naming it describe the same thing, so `--env` lets you skip writing the file: ```sh -spinloop fleet status --env qwen # the same row a one-node fleet file gives -spinloop fleet dashboard --env qwen # the tiled view, on one environment +spinloop status --env qwen # the same row a one-node fleet file gives +spinloop dashboard --env qwen # the tiled view, on one environment spinloop fleet logs --env qwen # its engine's log ``` @@ -66,7 +71,7 @@ you can see all of this working before setting up a single machine: cd examples/fleet-docker && cp .env.example .env docker compose up -d --build set -a && . ./.env && set +a -spinloop fleet status --fleet ./fleet.yaml +spinloop status --fleet ./fleet.yaml ``` ## `fleet.yaml` @@ -203,7 +208,7 @@ config-less start once this field exists. A `kind: remote` node's `start` is unaffected by any of this: what it serves is fixed at deploy time, not pushed at start time. -This does not apply to `spinloop fleet dashboard`'s `s` key, which still +This does not apply to `spinloop dashboard`'s `s` key, which still starts the selected node with a plain start, whatever the CLI's `fleet start` would resolve for it. @@ -469,7 +474,7 @@ silently missing whatever was down: ## The dashboard -`spinloop fleet dashboard` is that same board as a live view: one tile per +`spinloop dashboard` is that same board as a live view: one tile per node, repainted in place, each drawing exactly what `fleet metrics`' gauge format prints for the node — state and uptime, what it serves, the CPU/GPU/RAM gauges, the token counters — so the view and the one-shot command never @@ -480,8 +485,8 @@ is down is a tile that says why, and a node whose token reference resolves to nothing holds that reason for the life of the view: ```sh -spinloop fleet dashboard # ./fleet.yaml -spinloop fleet dashboard --fleet f.yaml # another fleet file +spinloop dashboard # ./fleet.yaml +spinloop dashboard --fleet f.yaml # another fleet file ``` | Key | Does | @@ -538,7 +543,7 @@ the control plane set and the node is re-read at once, which is what brings the relative `keep for …` figure onto the tile and detail screen at the node's next round rather than waiting out its full cadence. -Everything else in the view is `fleet status`/`metrics`/`logs` in place — it +Everything else in the view is `status`/`metrics`/`logs` in place — it is read-only apart from those four action keys. It needs a real terminal: a piped run is refused, and it says so by way of `fleet metrics --watch`, which is the streamable surface. @@ -552,7 +557,7 @@ keys the view answers to. `Esc` closes it and returns to the grid with the same node still selected. ```sh -spinloop fleet dashboard +spinloop dashboard # select a node, press Enter for its full metrics and log, Esc to go back ``` @@ -576,7 +581,7 @@ poll that simply ran late. ## Logs `spinloop fleet logs` prints what your engines actually said — the answer to the -question `fleet status` raises when it reports a node as `crashed`. +question `status` raises when it reports a node as `crashed`. ```sh spinloop fleet logs # the tail of every node's engine log diff --git a/docs/commands/serve.md b/docs/commands/serve.md index c345f19b..714fff6f 100644 --- a/docs/commands/serve.md +++ b/docs/commands/serve.md @@ -25,7 +25,7 @@ Run `serve` on a terminal and the engine runs under a full-screen view rather than forwarding its output: the engine's **metrics** above, its **log** below, and a footer naming the keys the view answers to. -- **Metrics** — the same facts the fleet dashboard's node detail screen shows +- **Metrics** — the same facts the dashboard's node detail screen shows for the same engine: state and uptime, what is served, last active, and the resource series — CPU, RAM, and each GPU's utilisation and memory — with every series drawn in both formats at once, each on one line: a gauge of @@ -407,7 +407,7 @@ Records are graded, which is what makes the level worth setting: | `warn` | Only rejected requests (401, a bad cursor), a slow shutdown escalating to a kill, and crashes | | `error` | Only crashes, failed starts, and requests that failed inside spinloop | -`--log-level warn` is the setting for a node a fleet polls: a `fleet status` +`--log-level warn` is the setting for a node a fleet polls: a `status` refresh every few seconds is a request each, and at the default level polling is quiet. `--log-level debug` is how to see the routine traffic; at `warn` the polling disappears and a wrong token still shows up. diff --git a/docs/commands/status.md b/docs/commands/status.md new file mode 100644 index 00000000..6bb870a7 --- /dev/null +++ b/docs/commands/status.md @@ -0,0 +1,64 @@ +# spinloop status + +What every engine you run is doing, one row each — whether that is a fleet of +machines, a single cloud environment, or both. + +```sh +spinloop status # the fleet.yaml in this directory +spinloop status --fleet ./cluster.yaml +spinloop status --env prod # one registered environment, no file needed +``` + +``` +NODE STATE SERVING +studio running llamacpp qwen3-27b (up 2h 14m) (active 3m ago) (1.40.0) +gpu-box stopped +prod running vllm org/model (active 12s ago) +dead unreachable dial tcp 198.51.100.9:4242: connect: connection refused +``` + +Nodes are queried concurrently, so the command takes as long as the slowest +one rather than all of them added up. A node that cannot be reached is a row +saying why, not a failure: one unreachable machine never blanks the rest, and +the command still succeeds. + +## Which target + +The same three ways every command that acts on a fleet takes one: + +| | target | +| --- | --- | +| `--env ` | one registered environment | +| `--fleet ` (`-f`) | that fleet file | +| neither | the `fleet.yaml` in the working directory | + +`--env` and `--fleet` name two different things, so passing both fails saying +so. With none of the three resolvable the command fails naming all of them — +it does not go looking for an engine on the machine you are sitting at. To +watch a local engine, [`spinloop serve`](serve.md) shows the one it runs; to +read it from elsewhere, run [`spinloop daemon`](serve.md#the-control-api---api-and-spinloop-daemon) +and name that machine in a `fleet.yaml`. + +## What a row says + +- **state** — `idle`, `running`, `stopped` or `crashed`, or the reason the + node did not answer. +- **serving** — the runner and model, then how long it has been up, how long + since it last did work, and the spinloop version of the daemon running it. +- **not ready** — the engine's process exists but it has not answered its own + health check, so it is not servable yet however long its uptime says. A + cloud environment whose endpoint the control plane reports unhealthy reads + the same way. + +Two things are deliberately not columns, because only one kind of node has +them: an environment's endpoint address, which is +[`spinloop remote env`](remote.md), and its retention deadline, which +[`spinloop fleet metrics`](fleet.md#metrics) and +[`spinloop dashboard`](dashboard.md) show. + +## See also + +- [`spinloop dashboard`](dashboard.md) — the same facts, live and interactive +- [`spinloop fleet`](fleet.md) — driving the nodes rather than reading them +- [`spinloop remote status`](remote.md) — one environment, with its address, + version and keep deadline in full diff --git a/examples/fleet-docker/README.md b/examples/fleet-docker/README.md index 9b89fd7c..50d6dee4 100644 --- a/examples/fleet-docker/README.md +++ b/examples/fleet-docker/README.md @@ -12,7 +12,7 @@ docker compose up -d --build # from this directory, with the token exported set -a && . ./.env && set +a -spinloop fleet status --fleet ./fleet.yaml +spinloop status --fleet ./fleet.yaml spinloop fleet start studio --fleet ./fleet.yaml spinloop fleet metrics -w --fleet ./fleet.yaml ``` @@ -75,15 +75,15 @@ spinloop harness open ./client/Spinloop --fleet ./fleet.yaml # A node that goes away: the row degrades, the rest keep reporting, exit 0. docker compose stop gpu-box -spinloop fleet status --fleet ./fleet.yaml +spinloop status --fleet ./fleet.yaml # A wrong token reads `unauthorized`, not `unreachable` — the boxes are up, # the credential is wrong. -FLEET_TOKEN=nope spinloop fleet status --fleet ./fleet.yaml +FLEET_TOKEN=nope spinloop status --fleet ./fleet.yaml # Kill an engine and watch the node report `crashed`, then bring it back. docker compose exec studio sh -c 'kill -9 $(pgrep imposter-go)' -spinloop fleet status --fleet ./fleet.yaml +spinloop status --fleet ./fleet.yaml spinloop fleet start studio --fleet ./fleet.yaml ``` diff --git a/examples/fleet-docker/compose.yaml b/examples/fleet-docker/compose.yaml index 78b4da5c..abcfefca 100644 --- a/examples/fleet-docker/compose.yaml +++ b/examples/fleet-docker/compose.yaml @@ -3,7 +3,7 @@ # # cp .env.example .env # docker compose up -d --build -# spinloop fleet status +# spinloop status # spinloop fleet metrics -w # # Every node carries the fleet's shared token: the daemon refuses to listen on diff --git a/examples/fleet-docker/run-tests.sh b/examples/fleet-docker/run-tests.sh index 97b45ce3..c7d77b27 100755 --- a/examples/fleet-docker/run-tests.sh +++ b/examples/fleet-docker/run-tests.sh @@ -128,7 +128,7 @@ container_state() { } ####################################### -# Dump what the containers are doing, for a wait that timed out. `fleet status` +# Dump what the containers are doing, for a wait that timed out. `status` # only reports that a port refused the connection; whether the container is # even up, and what its daemon said on the way down, is the part worth having. # Globals: @@ -177,7 +177,7 @@ fleet_with_stderr() { ####################################### node_state() { local name="$1" - fleet status | awk -v n="${name}" '$1 == n {print $2}' + status | awk -v n="${name}" '$1 == n {print $2}' } ####################################### @@ -195,13 +195,13 @@ wait_for_fleet() { # a `grep -q` that matches and exits first can leave the pipeline # reporting the writer's SIGPIPE, which reads here as "nothing # unreachable" — the opposite of what was found. - if [[ "$(fleet status)" != *unreachable* ]]; then + if [[ "$(status)" != *unreachable* ]]; then return 0 fi sleep 2 done echo "Error: the fleet did not become reachable in ${READY_TIMEOUT_SECS}s" >&2 - fleet status >&2 || true + status >&2 || true diagnose_fleet return 1 } @@ -297,7 +297,7 @@ cleanup() { echo echo "Stack left running (--keep). Try:" echo " cd ${HERE} && set -a && . ./.env && set +a" - echo " spinloop fleet status --fleet ${HERE}/fleet.yaml" + echo " spinloop status --fleet ${HERE}/fleet.yaml" echo "Tear down with: docker compose -f ${HERE}/compose.yaml down -v" return fi @@ -329,14 +329,14 @@ test_untold_node_cannot_start() { test_cold_start() { echo "Cold start: a usable fleet with nothing running" local out - out="$(fleet status)" + out="$(status)" assert_contains "status lists studio" "${out}" "studio" assert_contains "status lists gpu-box" "${out}" "gpu-box" assert_contains "status lists laptop" "${out}" "laptop" assert_equals "studio is idle before anything is started" \ "$(node_state studio)" "idle" # A fleet where nothing runs is still a working view, not an error. - fleet status >/dev/null + status >/dev/null assert_equals "status succeeds with nothing running" "$?" "0" } @@ -537,12 +537,12 @@ test_unreachable_node() { compose stop laptop local out - out="$(fleet status)" + out="$(status)" assert_contains "the stopped node reads unreachable" "${out}" "unreachable" assert_contains "a reason is shown" "${out}" "connect" assert_contains "other nodes still report" "${out}" "gpu-box" # The whole point: one bad node must not fail the command. - fleet status >/dev/null + status >/dev/null assert_equals "status still succeeds" "$?" "0" restart_node laptop || true @@ -554,7 +554,7 @@ test_unreachable_node() { test_unauthorized() { echo "A rejected token is distinguished from an unreachable node" local out - out="$(FLEET_TOKEN=definitely-not-the-token fleet status)" + out="$(FLEET_TOKEN=definitely-not-the-token status)" assert_contains "a bad token reads unauthorized" "${out}" "unauthorized" assert_not_contains "a bad token is not reported as unreachable" \ "$(echo "${out}" | grep '^studio')" "unreachable" diff --git a/examples/fleet-local/README.md b/examples/fleet-local/README.md index c747f508..0de20f56 100644 --- a/examples/fleet-local/README.md +++ b/examples/fleet-local/README.md @@ -90,7 +90,7 @@ Leave it running — under `launchd` or `systemd` for real use, or just in a terminal to try it. Then, from this directory: ```sh -spinloop fleet status # local: idle +spinloop status # local: idle spinloop fleet route # which node a launch would pick, changing nothing spinloop harness open -O # wear ./Spinloop, route, wake if needed, launch ``` diff --git a/examples/fleet-mixed/README.md b/examples/fleet-mixed/README.md index ee21eaca..f9950433 100644 --- a/examples/fleet-mixed/README.md +++ b/examples/fleet-mixed/README.md @@ -48,7 +48,7 @@ remote`](../../docs/commands/remote.md) — just one command for both. ### 3. Observe the whole fleet ```sh -spinloop fleet status # one row per node: the machine and the environments +spinloop status # one row per node: the machine and the environments spinloop fleet metrics -w # a live dashboard spinloop fleet start qwen # wake a sleeping environment from zero spinloop fleet stop gpu-box # stop the machine's engine diff --git a/examples/fleet-mixed/fleet.yaml b/examples/fleet-mixed/fleet.yaml index d530c5c0..15d88ade 100644 --- a/examples/fleet-mixed/fleet.yaml +++ b/examples/fleet-mixed/fleet.yaml @@ -2,7 +2,7 @@ # environments, observed side by side. The same command reaches every node, # whatever kind it is. # -# spinloop fleet status / metrics -w +# spinloop status / metrics -w # # A daemon node names the machine (host) and the variable holding its bearer # token (tokenEnv). A remote node names its environment — its node name *is* the diff --git a/examples/fleet-remote/README.md b/examples/fleet-remote/README.md index a780b2fc..cf1949e8 100644 --- a/examples/fleet-remote/README.md +++ b/examples/fleet-remote/README.md @@ -60,7 +60,7 @@ the full resolution order. From any machine your AWS credentials reach: ```sh -spinloop fleet status # one row per environment +spinloop status # one row per environment spinloop fleet metrics -w # a live dashboard spinloop fleet start qwen # wake a sleeping environment from zero spinloop fleet stop qwen # scale it back down diff --git a/examples/fleet-remote/fleet.yaml b/examples/fleet-remote/fleet.yaml index 2c27efad..ca410cda 100644 --- a/examples/fleet-remote/fleet.yaml +++ b/examples/fleet-remote/fleet.yaml @@ -1,6 +1,6 @@ # A fleet of `spinloop remote` environments, observed like machines. # -# spinloop fleet status # one row per environment: state and what it serves +# spinloop status # one row per environment: state and what it serves # spinloop fleet metrics -w # a live dashboard # spinloop fleet deploy --all # create both environments from this file # diff --git a/examples/fleet/README.md b/examples/fleet/README.md index 4e8a7985..7e00082c 100644 --- a/examples/fleet/README.md +++ b/examples/fleet/README.md @@ -19,7 +19,7 @@ automatically since the subdirectory's name matches the node's), so ```sh cp .env.example .env # fill in the fleet's shared token -spinloop fleet status +spinloop status spinloop fleet metrics -w spinloop fleet start gpu-box ``` diff --git a/examples/fleet/fleet.yaml b/examples/fleet/fleet.yaml index 7d3abe06..d2036dd5 100644 --- a/examples/fleet/fleet.yaml +++ b/examples/fleet/fleet.yaml @@ -1,6 +1,6 @@ # A fleet: the machines running `spinloop daemon`, and how to reach each one. # -# spinloop fleet status # one row per node +# spinloop status # one row per node # spinloop fleet metrics -w # a live dashboard # # This file holds no secrets. A node that needs a bearer token names the diff --git a/examples/gateway-docker/README.md b/examples/gateway-docker/README.md index 1bd6ded8..b1452917 100644 --- a/examples/gateway-docker/README.md +++ b/examples/gateway-docker/README.md @@ -20,7 +20,7 @@ curl -X POST -H "Authorization: Bearer $GATEWAY_TOKEN" \ http://127.0.0.1:4000/v1/chat/completions # and the fleet underneath it, the way spinloop fleet drives any fleet -spinloop fleet status --fleet ./fleet.yaml +spinloop status --fleet ./fleet.yaml spinloop fleet start node-b --fleet ./fleet.yaml ``` diff --git a/examples/gateway-docker/run-tests.sh b/examples/gateway-docker/run-tests.sh index 87833150..4dca6b1a 100755 --- a/examples/gateway-docker/run-tests.sh +++ b/examples/gateway-docker/run-tests.sh @@ -173,7 +173,7 @@ fleet_with_stderr() { ####################################### node_state() { local name="$1" - fleet status | awk -v n="${name}" '$1 == n {print $2}' + status | awk -v n="${name}" '$1 == n {print $2}' } ####################################### @@ -238,13 +238,13 @@ wait_for_fleet() { # a `grep -q` that matches and exits first can leave the pipeline # reporting the writer's SIGPIPE, which reads here as "nothing # unreachable" — the opposite of what was found. - if [[ "$(fleet status)" != *unreachable* ]]; then + if [[ "$(status)" != *unreachable* ]]; then return 0 fi sleep 2 done echo "Error: the fleet did not become reachable in ${READY_TIMEOUT_SECS}s" >&2 - fleet status >&2 || true + status >&2 || true diagnose_fleet return 1 } @@ -374,7 +374,7 @@ test_suggested_start_works() { else fail "fleet start node-a brings it up" "running" "$(node_state node-a)" fi - assert_contains "status shows what it serves" "$(fleet status)" "fake-model" + assert_contains "status shows what it serves" "$(status)" "fake-model" } ####################################### diff --git a/internal/fleet/config.go b/internal/fleet/config.go index a0454456..c9e04bc5 100644 --- a/internal/fleet/config.go +++ b/internal/fleet/config.go @@ -380,7 +380,7 @@ func Load(path string) (*Config, error) { if err != nil { if os.IsNotExist(err) { return nil, fmt.Errorf( - "no fleet file at %s: create one listing your nodes, or pass --fleet ", path) + "no fleet at %s: create one listing your nodes, or name a target — --fleet for a file, --env for a registered environment", path) } return nil, err } @@ -435,15 +435,10 @@ func ForEnvironment(name string) (*Config, error) { return nil, fmt.Errorf( "%q is not an environment name: an environment name is a plain identifier, with no path", name) } - path, err := remote.EnvConfigPath(name) - if err != nil { - return nil, err - } - if _, err := os.Stat(path); err != nil { - if os.IsNotExist(err) { - return nil, fmt.Errorf( - "environment %q is not registered: run `spinloop remote deploy --env %q` to create it", name, name) - } + // Loading it is the check: every environment resolves the one way, by + // name, and a name that resolves to nothing fails here rather than at the + // first control call. + if _, err := remote.LoadEnvironment(name, os.Getenv); err != nil { return nil, err } cfg := &Config{Nodes: []NodeConfig{{Name: name, Kind: KindRemote}}} diff --git a/internal/fleet/for_environment_test.go b/internal/fleet/for_environment_test.go index ab949429..bd47f74b 100644 --- a/internal/fleet/for_environment_test.go +++ b/internal/fleet/for_environment_test.go @@ -83,7 +83,7 @@ func TestForEnvironmentRejects(t *testing.T) { {"a nested path", "envs/prod", "plain identifier"}, {"a json file", "prod.json", "plain identifier"}, {"empty", "", "plain identifier"}, - {"unregistered", "nope", "is not registered"}, + {"unregistered", "nope", "remotes/nope/remote.json"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -98,16 +98,16 @@ func TestForEnvironmentRejects(t *testing.T) { } } -// An unregistered environment names how to create it, so the message is a -// repair rather than a report. +// An environment that resolves to nothing fails naming the file to create, so +// the message is a repair rather than a report. func TestForEnvironmentUnregisteredNamesTheFix(t *testing.T) { t.Setenv("SPINLOOP_CONFIG_DIR", t.TempDir()) _, err := ForEnvironment("nope") if err == nil { t.Fatal("want an error") } - if !strings.Contains(err.Error(), "spinloop remote deploy --env") { - t.Errorf("error %q does not name how to create the environment", err) + if !strings.Contains(err.Error(), "remotes/nope/remote.json") { + t.Errorf("error %q does not name the environment's registry path", err) } } diff --git a/openspec/changes/top-level-status-and-dashboard/design.md b/openspec/changes/top-level-status-and-dashboard/design.md index c4a7dad6..95517d55 100644 --- a/openspec/changes/top-level-status-and-dashboard/design.md +++ b/openspec/changes/top-level-status-and-dashboard/design.md @@ -12,10 +12,18 @@ See proposal.md — Why. The implementation-relevant current state: `fleetDashboardCmd` a flag block plus `dashModelFor` and `runDashProgram`. - `runRemoteStatus` (`cmd/spinloop/remote.go`) calls `remote.Status` directly and renders a key-value block. It already shares `statusFact` with the fleet - table — the facts converged, the layout did not. -- Two facts it renders are not on `daemon.StatusResponse` and so do not - survive a fan-out: the endpoint's **health** (`resp.Healthy`) and its - **base URL**. `statusFromRemote` drops both. + table — the facts converged, the layout did not. It also makes a second call + for the version, applies a Spinloop's `ENV` before resolving, and renders the + address and retention deadline: three things a fan-out does not do, which is + why it stays until `metrics` and `logs` force the same design. +- Of the facts `remote status` renders, **health** is already carried and + already drawn: `statusFromRemote` maps it onto `Ready`, and `servingText` + renders `ReadyNo` as `(not ready)`. That mapping arrived with the gateway's + wake-a-deployed-node work, which needs to know whether a woken environment + is serving before routing to it. +- The **base URL** is carried (decomposed into `Engine.Host/Port/Path`) but not + drawn; the **retention deadline** is on `metrics.Stats`, not + `daemon.StatusResponse`, so the status fan-out cannot reach it at all. - `movedTopLevelCommands` and `groupArgs`/`movedSubcommands` (`cmd/spinloop/commands.go`) are the two existing signposts: the first for a first word, the second for a group subcommand. @@ -56,45 +64,42 @@ levels (rejected — two entries in the tree for one behaviour is the thing this removes, and the house pattern for a moved command is a hard break that names the replacement). -**D2: Health and base URL ride on the status view, not on a node capability.** +**D2: Health and base URL need no new plumbing — both are already carried.** -`remote status` renders two facts a daemon has no answer for. The tempting -move is a `Healther` capability beside `Keeper`. It is the wrong shape: both -facts are already on the reply `remoteNode.Status` receives and simply are not -copied across, so a capability would be a second round trip to fetch what the -first one already returned. - -`daemon.StatusResponse` gains neither — it is the daemon's published contract -and a daemon cannot answer either question. Instead `fleet.NodeResult` carries -them alongside the status it already holds, filled by `statusFromRemote` and -left empty by a daemon node. The renderer shows them when present. +`remote status` renders facts a daemon has no answer for, and the tempting move +is a `Healther` capability beside `Keeper`, or a fleet-owned status type +wrapping the daemon's. Neither is needed. Health is already mapped onto `Ready` +and already drawn as `(not ready)`, so it costs nothing. The address and the +retention deadline are not drawn — deliberately: each is reported by a command +that exists to report it (`remote env` for the address, `metrics` and the +dashboard for the deadline), and a table read one row per node should not grow +a column only one node kind ever fills. This keeps the capability interfaces for what they are for — an operation a -node kind can or cannot *perform* (`Keep`, `StartWithProgress`) — rather than -a field one kind happens to know. +node kind can or cannot *perform* (`Keep`, `StartWithProgress`) — rather than a +field one kind happens to know. A capability here would be a second round trip +for a value the first call already returned. -**D3: A single-node target renders as the table, not as the old key-value -block.** +One inherited consequence is worth naming rather than rediscovering: `running` +in `select.go` skips a node reporting `Ready == ReadyNo`, so an unhealthy cloud +environment is not selected for routing. That follows from the health mapping, +which was settled when it landed; this change neither introduces nor alters it. -`remote status` printed a key-value block; `fleet status` prints a table. One -command cannot print both, and the table is the one that scales: it reads the -same for one target and for twelve, and it is what the dashboard's detail view -already echoes. +**D3: A single-node target renders as the table, whatever its node count.** -The key-value block's extra facts are not lost — D2 carries them — but they -appear as columns, not as lines. This is the one visible difference for an -operator who used `remote status`, and it is stated in the proposal. +`spinloop status --env prod` renders the one-row table, not a key-value block: +a command whose output shape depends on how many things it found is harder to +script against and harder to explain than one whose shape is fixed. It reads +the same for one target and for twelve, and it is what the dashboard's detail +view already echoes. -Alternative: switch layout on node count (rejected — a command whose output -shape depends on how many things it found is harder to script against and -harder to explain than one that does not). +`remote status`'s key-value block is untouched and still available — see the +proposal for why that command stays. -**D4: Three signposts, using both existing mechanisms.** +**D4: Two signposts, using the existing mechanism.** `fleet status` and `fleet dashboard` are group subcommands, so they go in -`movedSubcommands`, which `groupArgs` already reads. `remote status` is the -same shape, and the `remote` group gains the same `Args: groupArgs` the fleet -group has. No new mechanism. +`movedSubcommands`, which `groupArgs` already reads. No new mechanism. **D5: The no-target message is fixed here rather than left to the rename.** diff --git a/openspec/changes/top-level-status-and-dashboard/proposal.md b/openspec/changes/top-level-status-and-dashboard/proposal.md index a52f34aa..cfd8c871 100644 --- a/openspec/changes/top-level-status-and-dashboard/proposal.md +++ b/openspec/changes/top-level-status-and-dashboard/proposal.md @@ -30,9 +30,9 @@ this change mechanical and gives that design its own review. the way the fleet commands do — `--env `, `--fleet `, or the working directory's `fleet.yaml` — and renders exactly as its fleet-scoped spelling does today. -- **BREAKING** `spinloop fleet status`, `spinloop fleet dashboard` and - `spinloop remote status` are removed. Each fails naming the command that - replaced it, the way a moved command already does. +- **BREAKING** `spinloop fleet status` and `spinloop fleet dashboard` are + removed. Each fails naming the command that replaced it, the way a moved + command already does. - **BREAKING** With no target resolvable — no `--env`, no `--fleet`, no `./fleet.yaml` — the verbs fail rather than looking for an engine on this machine. `spinloop serve` already shows the engine it runs, and a machine @@ -40,9 +40,15 @@ this change mechanical and gives that design its own review. fleet file. The failure names all three ways to give a target, which today's message does not: it offers only `--fleet` and creating a file, though `--env` has been a target since it landed. -- What `remote status` reported that a node's status does not carry — the - endpoint's health and its address — is carried into the top-level verb, so - nothing an operator could see is lost with the command. +- `spinloop remote status` is **not** removed here. It does three things a + fan-out cannot: it makes a second control call for the version, it applies a + Spinloop's `ENV` instructions before resolving, and it renders the endpoint's + address and retention deadline. `remote metrics` does the same three, so the + design for carrying them belongs to the change that moves `metrics` and + `logs` — solving it once for all of them beats solving it twice. Until then + `spinloop status --env ` and `spinloop remote status --env ` + coexist, as `--env` and `--fleet` did through the change that introduced + them. - `fleet` keeps `route`, `start`, `stop`, `deploy`, `metrics` and `logs`; `remote` keeps everything else it has. Only `status` and `dashboard` move here. @@ -61,11 +67,6 @@ this change mechanical and gives that design its own review. - `fleet-config`: target resolution is stated for the top-level verbs as well as the fleet group, and the failure with no target names all three ways to give one. -- `remote-endpoint`: the `remote` group no longer has a `status` subcommand, - and what that subcommand reported — state, health, last-active — is reported - by the top-level `status` against the same environment. -- `remote-version-reporting`: the version an environment reports is read - through the top-level `status` rather than `remote status`. ## Impact @@ -73,12 +74,8 @@ this change mechanical and gives that design its own review. root, resolving through the existing `resolveFleetTarget`. - `cmd/spinloop/fleet.go`, `fleet_dashboard.go`: lose their command wrappers, keep their renderers and the dashboard model. -- `cmd/spinloop/remote.go`: loses `remoteStatusCmd` and `runRemoteStatus`; the - health and address facts it rendered move into the shared status view. - `cmd/spinloop/commands.go`: the two verbs registered, the two fleet - subcommands unregistered, and the moved spellings signposted — `fleet - status`, `fleet dashboard` and `remote status` each naming their - replacement. + subcommands unregistered, and both moved spellings signposted. - `internal/fleet/config.go`: the no-target failure names `--env` too. - `docs/commands/`: pages for the two verbs; `fleet.md` and `remote.md` point at them. diff --git a/openspec/changes/top-level-status-and-dashboard/specs/fleet-client/spec.md b/openspec/changes/top-level-status-and-dashboard/specs/fleet-client/spec.md index c7dbbba4..e3109697 100644 --- a/openspec/changes/top-level-status-and-dashboard/specs/fleet-client/spec.md +++ b/openspec/changes/top-level-status-and-dashboard/specs/fleet-client/spec.md @@ -4,7 +4,9 @@ `spinloop status` SHALL query every node in the resolved target and render one row per node: the node name, its engine state (`idle`/`running`/`stopped`/`crashed`), what it is serving (runner and model when known), the spinloop version of the daemon on that node, and its reachability. Nodes SHALL be queried concurrently so the command's latency is that of the slowest reachable node, not their sum. -The target SHALL resolve as it does for every other command that acts on a fleet — a named environment, a named fleet file, or the working directory's fleet file — so one environment, a fleet, and a fleet holding that same environment are all read by the one command. A cloud environment SHALL additionally report the endpoint's health and its address, which a daemon node does not carry; a target that reports neither SHALL omit them rather than render them empty. +The target SHALL resolve as it does for every other command that acts on a fleet — a named environment, a named fleet file, or the working directory's fleet file — so one environment, a fleet, and a fleet holding that same environment are all read by the one command, rendered identically however the target was named. + +A node's health SHALL be reported as the readiness mark every node carries, so a cloud environment whose endpoint is unhealthy reads the same as a daemon node whose engine has not answered its health check. A field only one node kind fills — an endpoint's address, a retention deadline — SHALL NOT be a column of a table meant to be scanned one row per node. A node SHALL also report how long it has been since its engine last did work, taken from the activity its daemon tracks — "which of my nodes is doing nothing?" is a question a fleet view exists to answer, and the daemon already knows. That figure SHALL NOT be labelled in a way that collides with the `idle` engine state, which means something different. A node whose daemon reports no activity yet SHALL omit the figure rather than imply an engine has sat unused since it started. @@ -16,8 +18,15 @@ A node SHALL also report how long it has been since its engine last did work, ta #### Scenario: A named environment is read by the same command - **WHEN** `spinloop status --env prod` runs -- **THEN** that environment's state, health and address are reported, and no - fleet file is required +- **THEN** that environment's row is rendered with no fleet file required, and + it reads the same as the row for the same environment named in a fleet file + +#### Scenario: An unhealthy endpoint reads as not ready + +- **WHEN** `spinloop status --env prod` runs and the control plane reports the + endpoint unhealthy +- **THEN** the row carries the same not-ready mark a daemon node carries when + its engine has not answered its health check #### Scenario: A node reports how long since it last did work diff --git a/openspec/changes/top-level-status-and-dashboard/specs/remote-endpoint/spec.md b/openspec/changes/top-level-status-and-dashboard/specs/remote-endpoint/spec.md deleted file mode 100644 index 52f9aa2b..00000000 --- a/openspec/changes/top-level-status-and-dashboard/specs/remote-endpoint/spec.md +++ /dev/null @@ -1,215 +0,0 @@ -## MODIFIED Requirements - -### Requirement: Remote command group - -The system SHALL provide a `remote` command group with the subcommands -`bootstrap`, `bake`, `auth`, `start`, `stop`, `restart`, `deploy`, -`ls`, `metrics`, and `keep`. `start`, `stop`, `restart`, `metrics` and -`deploy` each take an optional Spinloop path: -`start` SHALL boot the endpoint and block until it is serving, then perform a -quick TCP probe of the inference endpoint — if the probe fails, a warning is -printed to stderr explaining the network mismatch (see the Remote Start Probe -specification) — and finally print the base URL and API key as shell exports; -`start` SHALL also accept a `--keep DURATION` flag that sets the instance -retention deadline to `now + DURATION`, preventing the idle sweep from -terminating it before that time (see the Remote Keep specification); -`stop` SHALL stop it immediately rather than waiting for its idle timer; -`restart` SHALL stop the endpoint in the manner of a pause — without -terminating it, so its boot disk, its weights and its stable address are -preserved — and SHALL immediately start it again, blocking until it is serving -and reporting progress as `start` does (see the Reporting a start in progress -specification); `restart` SHALL accept a `--force` flag with a `-F` short form -that, when set, performs the stop without first asking the engine to shut down -(see the Endpoint Lifecycle specification for forced stops); -`keep` SHALL set the `Retain-Until` tag on the environment's instance for the -given duration, without starting or stopping the instance (see the Remote Keep -specification); `metrics` SHALL report instance state, token usage, resource -consumption, and GPU information for a running instance; `deploy` SHALL set -what the endpoint serves. `ls` SHALL list the registered remote environments -(see the Remote Environments specification). `bootstrap` SHALL stand up the -account-level AWS control plane (once per account) by obtaining and driving the -CDK project, and takes its own flags rather than a Spinloop path (see the -Endpoint Provisioning specification). `bake` SHALL start an AMI bake for each -runner named, and takes runner names rather than a Spinloop path (see the -Endpoint Provisioning specification). `auth` SHALL store, report, and clear the -long-lived control-plane credential, and takes its own flags rather than a -Spinloop path (see the Remote Auth specification). An unrecognised subcommand -SHALL fail naming the accepted ones. - -#### Scenario: Starting the endpoint - -- **WHEN** the user runs `spinloop remote start` and the endpoint reports ready -- **THEN** the base URL and API key are printed as `export` lines - -#### Scenario: Starting warns when the network is not admitted - -- **WHEN** the user runs `spinloop remote start` and the endpoint reports ready - but the TCP probe to the inference port fails -- **THEN** a warning is printed to stderr with a remediation command, and the - command still exits 0 - -#### Scenario: Starting with a keep flag - -- **WHEN** the user runs `spinloop remote start --keep 4h` and the endpoint reports ready -- **THEN** the base URL and API key are printed as `export` lines, and the - instance retention deadline is set to 4 hours from now - -#### Scenario: Waiting through a cold start - -- **WHEN** the endpoint reports that it is still starting -- **THEN** the command waits and retries until it is ready or the timeout - passes, rather than failing on the first attempt - -#### Scenario: Restarting the endpoint - -- **WHEN** the user runs `spinloop remote restart` for a running environment and - the endpoint reports ready again -- **THEN** the instance was stopped and re-woken without being terminated, the - command blocked until the model was serving again, and the environment's - address is the one its configuration records - -#### Scenario: Forcing a restart skips the engine stop - -- **WHEN** the user runs `spinloop remote restart --force` (or `-F`) -- **THEN** the instance is stopped without the engine being asked to shut down - first, and the command then blocks until the model is serving again - -#### Scenario: Restarting a stopped endpoint starts it - -- **WHEN** the user runs `spinloop remote restart` for an environment whose instance is already stopped -- **THEN** the instance is re-woken rather than replaced, and the command blocks - until the model is serving again, as with a plain start - -#### Scenario: A failed re-wake says how to recover - -- **WHEN** the stop half of a restart has taken effect but the wake fails -- **THEN** the command fails saying the instance is stopped and that - `spinloop remote start` will bring it back - -#### Scenario: Listing environments - -- **WHEN** the user runs `spinloop remote ls` -- **THEN** the registered environments are listed rather than any endpoint being - contacted - -#### Scenario: Setting a keep deadline - -- **WHEN** the user runs `spinloop remote keep 2h` -- **THEN** the instance retention tag is set and the deadline is reported - -#### Scenario: Metrics reports instance figures - -- **WHEN** the user runs `spinloop remote metrics` with a running instance -- **THEN** token counts, resource usage, and GPU information are displayed - -#### Scenario: Bootstrap is a recognised subcommand - -- **WHEN** the user runs `spinloop remote bootstrap` -- **THEN** the command is dispatched to the provisioning flow rather than - reported as unknown - -#### Scenario: Bake is a recognised subcommand - -- **WHEN** the user runs `spinloop remote bake llamacpp` -- **THEN** the command is dispatched to the bake flow rather than - reported as unknown - -#### Scenario: Auth is a recognised subcommand - -- **WHEN** the user runs `spinloop remote auth` -- **THEN** the command is dispatched to the credential store, report, and clear - flow rather than reported as unknown - -#### Scenario: Unknown subcommand - -- **WHEN** the user runs `spinloop remote frobnicate` -- **THEN** the command fails listing the accepted subcommands, which include - `bootstrap`, `bake`, `metrics`, and `keep` - -## REMOVED Requirements - -### Requirement: Status reports when the endpoint last did work - -**Reason**: `spinloop remote status` is removed — one command reads an -endpoint's state, whether it is named as an environment or as a fleet node. -The behaviour is unchanged and is restated below under a name that does not -carry the removed command's spelling. - -**Migration**: `spinloop status --env `. - -### Requirement: Status degrades when activity cannot be read - -**Reason**: As above — the requirement describes `spinloop remote status`, -which no longer exists. Restated below, unchanged in substance. - -**Migration**: `spinloop status --env `. - -## ADDED Requirements - -### Requirement: An environment reports when it last did work - -`spinloop status --env ` SHALL report how long it has been since the endpoint's -engine last did any work, alongside the instance state and health it reports -already. The figure SHALL come from the activity the on-instance daemon -tracks, not from a measurement the control plane makes itself — one answer, -derived on the box, however it is asked for. - -The figure SHALL be labelled "last active", matching the wording and duration -formatting used everywhere else this fact appears, so the same fact reads the -same way in every command. - -Collecting it SHALL NOT make `status` slower than its health check already -makes it: the daemon SHALL be asked in parallel with the health check rather -than after it. Nor SHALL it introduce a side effect — `status` SHALL remain a -read, and SHALL still perform no TCP probe. - -#### Scenario: A running endpoint reports its last activity - -- **WHEN** the user runs `spinloop status --env ` against a running endpoint - whose engine has served work -- **THEN** the output reports how long ago that work happened, labelled "last - active", beside the state and health lines - -#### Scenario: Status stays a read - -- **WHEN** the user runs `spinloop status --env ` -- **THEN** nothing is started, stopped or probed in order to obtain the - last-active figure - -### Requirement: An environment's status degrades when activity cannot be read - -`spinloop status --env ` SHALL omit the last-active figure rather than fail, -report zero, or imply inactivity, whenever the figure cannot be obtained. That -covers an endpoint whose engine has not yet done any work, a daemon that -cannot be reached or answers unrecognisably, and an instance that is not -running — reaching the daemon needs a running box, so a stopped or undeployed -environment has nothing to report about its engine. - -A failure to read the activity SHALL NOT affect the rest of the report: the -state and health lines SHALL be exactly what they are today, and the command -SHALL still succeed. - -#### Scenario: A stopped instance reports no activity figure - -- **WHEN** the user runs `spinloop status --env ` and the instance is stopped or - undeployed -- **THEN** the output reports the state as it does today and shows no - last-active figure - -#### Scenario: An unreachable daemon does not spoil the report - -- **WHEN** the endpoint is running but its daemon cannot be reached -- **THEN** the state and health lines are reported as they are today, no - last-active figure is shown, and the command succeeds - -#### Scenario: An engine that has done nothing yet - -- **WHEN** the endpoint is running and its daemon reports no last-active time -- **THEN** no last-active figure is shown, rather than one implying the engine - has been quiet since it started - -#### Scenario: The remote spelling names its replacement - -- **WHEN** the operator runs `spinloop remote status` -- **THEN** it fails naming `spinloop status --env ` as the command that - replaced it diff --git a/openspec/changes/top-level-status-and-dashboard/specs/remote-version-reporting/spec.md b/openspec/changes/top-level-status-and-dashboard/specs/remote-version-reporting/spec.md deleted file mode 100644 index d575784d..00000000 --- a/openspec/changes/top-level-status-and-dashboard/specs/remote-version-reporting/spec.md +++ /dev/null @@ -1,52 +0,0 @@ -## REMOVED Requirements - -### Requirement: Remote status shows version - -**Reason**: `spinloop remote status` is removed; the version it displayed is -displayed by the command that replaced it. Restated below under a name that -does not carry the removed spelling. - -**Migration**: `spinloop status --env `. - -### Requirement: Fleet status shows version - -**Reason**: `spinloop fleet status` is removed in favour of the top-level -`spinloop status`, which serves every target kind. The per-node version -behaviour is unchanged and is restated below. - -**Migration**: `spinloop status`. - -## ADDED Requirements - -### Requirement: An environment's status shows version - -`spinloop status --env ` SHALL display the spinloop version running on the remote instance alongside its existing state, health, and base URL fields. - -#### Scenario: Version is shown when the instance is running - -- **WHEN** the user runs `spinloop status --env ` against a running instance -- **THEN** the output includes a `version` line with the spinloop version string (e.g. `version: 1.16.0`) - -#### Scenario: Version is unavailable when the instance is stopped - -- **WHEN** the user runs `spinloop status --env ` against a stopped instance -- **THEN** the output omits the version line, since the daemon is not reachable - -### Requirement: Status shows version per node - -`spinloop status` SHALL display the spinloop version for each node alongside its existing state and serving columns, read from the daemon's `/v1/status` response. - -#### Scenario: Version is shown per node - -- **WHEN** `spinloop status` runs against a fleet of running nodes -- **THEN** each node's row includes the spinloop version string - -#### Scenario: Version is omitted for unreachable nodes - -- **WHEN** a node's daemon is unreachable -- **THEN** that node's row shows its failure outcome without a version - -#### Scenario: Versions differ across nodes - -- **WHEN** nodes in the fleet run different spinloop versions -- **THEN** each node's row shows its own version, making the difference visible diff --git a/openspec/changes/top-level-status-and-dashboard/tasks.md b/openspec/changes/top-level-status-and-dashboard/tasks.md index e454ccc5..3880814d 100644 --- a/openspec/changes/top-level-status-and-dashboard/tasks.md +++ b/openspec/changes/top-level-status-and-dashboard/tasks.md @@ -1,65 +1,64 @@ -## 1. Carrying the cloud-only facts +## 1. The cloud-only facts -- [ ] 1.1 Add the endpoint's health and base URL to `fleet.NodeResult`, filled - by `statusFromRemote` from the reply it already receives and left empty - by a daemon node (design D2). Verify a unit test over both node kinds: - the cloud node carries both, the daemon node neither. -- [ ] 1.2 Render them in the status table when present, omitted when not. - Verify the same environment read through a fleet file shows what - `remote status` shows today for health and address. +- [x] 1.1 Verify health already renders: a cloud node reporting unhealthy + carries the same not-ready mark a daemon node does, through the `Ready` + mapping and `servingText`. No code change expected — assert it with a + test so the guarantee is held rather than assumed. +- [x] 1.2 Verify the address and the retention deadline are reachable without + the removed command, and say where in the docs (design D2): the address + from `remote env`, the deadline from `metrics` and the dashboard. ## 2. The verbs -- [ ] 2.1 Add `cmd/spinloop/status.go`: a root-registered `status` taking +- [x] 2.1 Add `cmd/spinloop/status.go`: a root-registered `status` taking `--env`/`--fleet`, resolving through `resolveFleetTarget` and rendering through `renderFleetStatus`. Verify `spinloop status --env ` and `spinloop status --fleet ` both render, and that naming both fails with the shared message. -- [ ] 2.2 Add `cmd/spinloop/dashboard.go` the same way, over `dashModelFor` +- [x] 2.2 Add `cmd/spinloop/dashboard.go` the same way, over `dashModelFor` and `runDashProgram`. Verify `spinloop dashboard --env ` opens on one panel in a directory with no fleet file, and that a non-terminal invocation is refused as before. -- [ ] 2.3 Register both at the root and carry over the completion the fleet +- [x] 2.3 Register both at the root and carry over the completion the fleet spellings had (`--fleet` from files, `--env` from registered environments). Verify the completion test covers both new commands. ## 3. Removing the old spellings -- [ ] 3.1 Delete `fleetStatusCmd` and `fleetDashboardCmd` and unregister them, +- [x] 3.1 Delete `fleetStatusCmd` and `fleetDashboardCmd` and unregister them, keeping `renderFleetStatus`, `dashModelFor` and `runDashProgram` where they are. Verify `spinloop fleet --help` lists neither. -- [ ] 3.2 Delete `remoteStatusCmd` and `runRemoteStatus` and unregister them. - Verify `spinloop remote --help` no longer lists `status`, and that - nothing else referenced the removed functions. -- [ ] 3.3 Signpost all three moved spellings (design D4): `fleet status` and - `fleet dashboard` through the existing `movedSubcommands`, and - `remote status` by giving the `remote` group the same `Args: groupArgs` - the fleet group has. Verify each names its replacement, and that an - unknown subcommand in either group still gets cobra's own error. +- [x] 3.2 Verify `spinloop remote status` is untouched and still works: it + makes a second call for the version, applies a Spinloop's `ENV`, and + renders the address and retention deadline, none of which a fan-out + does. It goes with `metrics` and `logs`, not here. +- [x] 3.3 Signpost both moved spellings through the existing + `movedSubcommands` (design D4). Verify each names its replacement, and + that an unknown `fleet` subcommand still gets cobra's own error. ## 4. The no-target message -- [ ] 4.1 Change `fleet.Resolve`'s failure to name the expected path, +- [x] 4.1 Change `fleet.Resolve`'s failure to name the expected path, `--fleet ` and `--env ` (design D5). Verify a test asserts all three appear, and that the existing tests asserting on that message still pass or are updated to the fuller wording. -- [ ] 4.2 Verify no implicit local target: a read verb with nothing resolvable +- [x] 4.2 Verify no implicit local target: a read verb with nothing resolvable fails as above even on a machine whose daemon is answering on its default port. ## 5. Consumers, docs and verification -- [ ] 5.1 Update every invocation of the three moved spellings across `docs/`, +- [x] 5.1 Update every invocation of the two moved spellings across `docs/`, `README.md` and `examples/` — including the CI-run `examples/fleet-docker/run-tests.sh` and `examples/gateway-docker/run-tests.sh` if they use them. Verify no consumer still invokes a removed spelling. -- [ ] 5.2 Add `docs/commands/status.md` and `docs/commands/dashboard.md`, and +- [x] 5.2 Add `docs/commands/status.md` and `docs/commands/dashboard.md`, and point `fleet.md` and `remote.md` at them. Verify `docs/README.md`'s command table lists both. -- [ ] 5.3 Run `gofmt -l .` (expect no output), `go vet ./...` and +- [x] 5.3 Run `gofmt -l .` (expect no output), `go vet ./...` and `go test ./... -cover`, confirming total coverage is unchanged and still >= 80%. -- [ ] 5.4 Verify `metrics` and `logs` are untouched: no file under +- [x] 5.4 Verify `metrics` and `logs` are untouched: no file under `cmd/spinloop/` that implements them changed except where a shared renderer moved, and `fleet metrics`/`fleet logs`/`remote metrics`/`remote logs` all still work. From b22e4864d92d4eff56b5e1516645464f16cdc4ea Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Tue, 15 Sep 2026 01:57:06 +0100 Subject: [PATCH 3/3] fix(examples): give the integration scripts a status wrapper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two run-tests.sh scripts call a fleet() shell helper that prefixes `spinloop fleet`, so `fleet status` was a function call, not a command line. Moving status to the top level rewrote those call sites to a bare `status`, which is no function at all — the fleet-docker suite failed on it. Each script gains a status() wrapper beside fleet(), so the call sites read the same and reach the command that replaced the subcommand. --- examples/fleet-docker/run-tests.sh | 12 ++++++++++++ examples/fleet-mixed/fleet.yaml | 2 +- examples/fleet-remote/fleet.yaml | 2 +- examples/fleet/fleet.yaml | 2 +- examples/gateway-docker/run-tests.sh | 12 ++++++++++++ 5 files changed, 27 insertions(+), 3 deletions(-) diff --git a/examples/fleet-docker/run-tests.sh b/examples/fleet-docker/run-tests.sh index c7d77b27..5511d17b 100755 --- a/examples/fleet-docker/run-tests.sh +++ b/examples/fleet-docker/run-tests.sh @@ -154,6 +154,18 @@ fleet() { "${SPINLOOP_BIN}" fleet "$@" --fleet "${HERE}/fleet.yaml" 2>/dev/null } +####################################### +# `spinloop status` against this example's fleet file. status is a top-level +# verb rather than a fleet subcommand, so it needs its own wrapper. +# Globals: +# SPINLOOP_BIN, HERE +# Outputs: +# The command's stdout; stderr is discarded so assertions read cleanly. +####################################### +status() { + "${SPINLOOP_BIN}" status --fleet "${HERE}/fleet.yaml" 2>/dev/null +} + ####################################### # As fleet(), but merging stderr — for assertions about error messages, which # the CLI writes to stderr. diff --git a/examples/fleet-mixed/fleet.yaml b/examples/fleet-mixed/fleet.yaml index 15d88ade..ebfd0bba 100644 --- a/examples/fleet-mixed/fleet.yaml +++ b/examples/fleet-mixed/fleet.yaml @@ -2,7 +2,7 @@ # environments, observed side by side. The same command reaches every node, # whatever kind it is. # -# spinloop status / metrics -w +# spinloop status, or spinloop fleet metrics -w # # A daemon node names the machine (host) and the variable holding its bearer # token (tokenEnv). A remote node names its environment — its node name *is* the diff --git a/examples/fleet-remote/fleet.yaml b/examples/fleet-remote/fleet.yaml index ca410cda..6eafd174 100644 --- a/examples/fleet-remote/fleet.yaml +++ b/examples/fleet-remote/fleet.yaml @@ -1,6 +1,6 @@ # A fleet of `spinloop remote` environments, observed like machines. # -# spinloop status # one row per environment: state and what it serves +# spinloop status # one row per environment: state and what it serves # spinloop fleet metrics -w # a live dashboard # spinloop fleet deploy --all # create both environments from this file # diff --git a/examples/fleet/fleet.yaml b/examples/fleet/fleet.yaml index d2036dd5..eeb3d5d4 100644 --- a/examples/fleet/fleet.yaml +++ b/examples/fleet/fleet.yaml @@ -1,6 +1,6 @@ # A fleet: the machines running `spinloop daemon`, and how to reach each one. # -# spinloop status # one row per node +# spinloop status # one row per node # spinloop fleet metrics -w # a live dashboard # # This file holds no secrets. A node that needs a bearer token names the diff --git a/examples/gateway-docker/run-tests.sh b/examples/gateway-docker/run-tests.sh index 4dca6b1a..f2483e20 100755 --- a/examples/gateway-docker/run-tests.sh +++ b/examples/gateway-docker/run-tests.sh @@ -157,6 +157,18 @@ fleet() { "${SPINLOOP_BIN}" fleet "$@" --fleet "${HERE}/fleet.yaml" 2>/dev/null } +####################################### +# `spinloop status` against this example's fleet file. status is a top-level +# verb rather than a fleet subcommand, so it needs its own wrapper. +# Globals: +# SPINLOOP_BIN, HERE +# Outputs: +# The command's stdout; stderr is discarded so assertions read cleanly. +####################################### +status() { + "${SPINLOOP_BIN}" status --fleet "${HERE}/fleet.yaml" 2>/dev/null +} + ####################################### # As fleet(), but merging stderr — for assertions about error messages. # Globals: