Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,14 +63,14 @@ want one — starting when you ask, and still up for your next session.
spinloop daemon # ready on :4242; no model runs until you ask

# on the machine you work at — fleet.yaml names the nodes
spinloop fleet status # one row per node: state, and what it serves
spinloop fleet dashboard # the same fleet, as a board you leave open
spinloop status # one row per node: state, and what it serves
spinloop dashboard # the same fleet, as a board you leave open
```

Start the whole fleet with `spinloop up` or just one with `spinloop fleet start gpu-box` — this gets you the model from step 1 running on a machine across the room, or over Tailscale.

<p align="center">
<img src="docs/img/fleet_dashboard.png" alt="The spinloop fleet dashboard: four nodes serving Qwen3.8-27B under llama.cpp, and a fifth cloud node not yet deployed" width="900">
<img src="docs/img/fleet_dashboard.png" alt="The spinloop dashboard: four nodes serving Qwen3.8-27B under llama.cpp, and a fifth cloud node not yet deployed" width="900">
</p>

> In this screenshot, five nodes are configured and four are up, each serving the same
Expand Down Expand Up @@ -537,8 +537,8 @@ that node's model needs a key of its own, name it with `engineTokenEnv`: driving
a machine and talking to the model on it are separate credentials.

```sh
spinloop fleet status # one row per node: state and what it serves
spinloop fleet dashboard # the interactive tiled view — watch it, drive it
spinloop status # one row per node: state and what it serves
spinloop dashboard # the interactive tiled view — watch it, drive it
spinloop fleet start gpu-box # start one node's engine
```

Expand Down Expand Up @@ -596,7 +596,7 @@ engine — in about a minute:
cd examples/fleet-docker && cp .env.example .env
docker compose up -d --build
set -a && . ./.env && set +a
spinloop fleet status --fleet ./fleet.yaml
spinloop status --fleet ./fleet.yaml
```

Only one machine? A fleet of one is still worth it —
Expand Down
9 changes: 6 additions & 3 deletions cmd/spinloop/commands.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ has been.`,
unaliasCmd(),
serveCmd(),
upCmd(),
statusCmd(),
dashboardCmd(),
codeCmd(),
daemonCmd(),
gatewayCmd(),
Expand Down Expand Up @@ -344,7 +346,9 @@ func groupArgs(c *cobra.Command, args []string) error {
// of movedTopLevelCommands: the same signpost, for a spelling that was two
// words rather than one.
var movedSubcommands = map[string]string{
"fleet harness": "code --fleet <path>",
"fleet harness": "code --fleet <path>",
"fleet status": "status",
"fleet dashboard": "dashboard",
}

// fleetCmd builds the fleet parent and its subcommands. The parent does
Expand All @@ -366,10 +370,8 @@ error — only a problem with the fleet file itself fails a command.`,
RunE: groupFallback,
}
fleet.AddCommand(
fleetStatusCmd(),
fleetMetricsCmd(),
fleetLogsCmd(),
fleetDashboardCmd(),
fleetRouteCmd(),
fleetStartCmd(),
fleetStopCmd(),
Expand All @@ -389,6 +391,7 @@ the same Spinloop. The endpoint's URLs come from the Spinloop's REMOTE — a bar
name selects an environment under ~/.config/spinloop/remotes/<name>/, a path
names a file — falling back to the default environment. Each subcommand's
--help says what that step does.`,
Args: groupArgs,
SilenceErrors: true,
SilenceUsage: true,
RunE: groupFallback,
Expand Down
55 changes: 55 additions & 0 deletions cmd/spinloop/dashboard.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
// `spinloop dashboard`: the interactive board over whatever the target names —
// a fleet, or a single registered environment, which opens as a board of one.
// The model and the renderers live in dashboard_model.go and
// dashboard_render.go, and the program in fleet_dashboard.go; this is the
// command.

package main

import (
"github.com/spf13/cobra"
)

func dashboardCmd() *cobra.Command {
var path, envName string
c := &cobra.Command{
Use: "dashboard",
Short: "watch the engines in an interactive tiled view",
Long: `An interactive live view of the target: a tile per node, each drawing
what the gauge format of fleet metrics prints — state, what it serves, the
resource gauges, the token counters — repainted on an interval. A single
registered environment (--env) opens as a board of one.

The view is read-only apart from four keys: s starts the selected node, k
keeps a remote environment for a duration you type — it asks how long,
pre-filled with 4h, and reports the deadline the control plane set when the
keep is done — a abandons a start still in flight on it (the wait ends, the
node is free again — a wake the cloud is carrying goes on), x stops it after
a confirmation. The arrow keys move the selection, r forces a refresh, q or
Ctrl+C leaves. The keep key shows only for a node that can be kept — a remote
environment — and a kept environment's tile and detail view carry its
deadline beside the last-active line, whatever the engine's state.

A node that cannot be reached is still a tile, showing why, and a node whose
token reference is unresolvable holds its reason for the life of the view.
The board needs a terminal; to stream the metrics into a pipe, use fleet
metrics --watch instead.`,
Args: cobra.NoArgs,
SilenceErrors: true,
SilenceUsage: true,
RunE: func(c *cobra.Command, _ []string) error {
resolve(c)
return runFleetDashboard(fleetTarget{envName: envName, fleetPath: path})
},
}
fs := c.Flags()
fs.StringVarP(&path, "fleet", "f", "", fleetFileUsage)
fs.StringVar(&envName, "env", "", envFlagTargetUsage)
c.ValidArgsFunction = noPositionals
compRegister(c, "fleet", compFiles)
compRegister(c, "env", compEnvs)
return c
}

// cmdDashboard runs the command through the tree — the seam the suite calls.
func cmdDashboard(args []string) error { return execCmd(dashboardCmd(), args) }
30 changes: 0 additions & 30 deletions cmd/spinloop/fleet.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,36 +37,6 @@ func cmdFleet(args []string) error {
// fleetFileFlag is the --fleet flag's help, shared by every fleet subcommand.
const fleetFileUsage = "path to the fleet file (default ./fleet.yaml)"

// fleetStatusCmd reports every node's engine state, one row per node. A node
// that cannot be reached is a row, not a failure: the rest of the fleet still
// renders and the command still succeeds.
func fleetStatusCmd() *cobra.Command {
var path, envName string
c := &cobra.Command{
Use: "status",
Short: "report every node's engine state",
Args: cobra.ArbitraryArgs,
SilenceErrors: true,
SilenceUsage: true,
RunE: func(c *cobra.Command, _ []string) error {
resolve(c)
cfg, err := resolveFleetTarget(fleetTarget{envName: envName, fleetPath: path})
if err != nil {
return err
}
results := cfg.FanOut(context.Background(), fleet.StatusCall)
renderFleetStatus(os.Stdout, results)
return nil
},
}
c.Flags().StringVarP(&path, "fleet", "f", "", fleetFileUsage)
c.Flags().StringVar(&envName, "env", "", envFlagTargetUsage)
c.ValidArgsFunction = noPositionals
compRegister(c, "fleet", compFiles)
compRegister(c, "env", compEnvs)
return c
}

// renderFleetStatus writes the status table: node, state, what it serves, and
// the reason when a node did not answer.
func renderFleetStatus(w io.Writer, results []fleet.NodeResult) {
Expand Down
42 changes: 0 additions & 42 deletions cmd/spinloop/fleet_dashboard.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,52 +11,10 @@ import (
"os"

tea "github.com/charmbracelet/bubbletea"
"github.com/spf13/cobra"
"github.com/spinloop-ai/spinloop/internal/fleet"
"golang.org/x/term"
)

// fleetDashboardCmd builds the `fleet dashboard` subcommand.
func fleetDashboardCmd() *cobra.Command {
var path, envName string
c := &cobra.Command{
Use: "dashboard",
Short: "watch the fleet in an interactive tiled view",
Long: `An interactive live view of the fleet: a tile per node, each drawing
what the gauge format of fleet metrics prints — state, what it serves, the
resource gauges, the token counters — repainted on an interval.

The view is read-only apart from four keys: s starts the selected node, k
keeps a remote environment for a duration you type — it asks how long,
pre-filled with 4h, and reports the deadline the control plane set when the
keep is done — a abandons a start still in flight on it (the wait ends, the
node is free again — a wake the cloud is carrying goes on), x stops it after
a confirmation. The arrow keys move the selection, r forces a refresh, q or
Ctrl+C leaves. The keep key shows only for a node that can be kept — a remote
environment — and a kept environment's tile and detail view carry its
deadline beside the last-active line, whatever the engine's state.

A node that cannot be reached is still a tile, showing why, and a node whose
token reference is unresolvable holds its reason for the life of the view.
The board needs a terminal; to stream the metrics into a pipe, use fleet
metrics --watch instead.`,
Args: cobra.ArbitraryArgs,
SilenceErrors: true,
SilenceUsage: true,
RunE: func(c *cobra.Command, _ []string) error {
resolve(c)
return runFleetDashboard(fleetTarget{envName: envName, fleetPath: path})
},
}
fs := c.Flags()
fs.StringVarP(&path, "fleet", "f", "", fleetFileUsage)
fs.StringVar(&envName, "env", "", envFlagTargetUsage)
c.ValidArgsFunction = noPositionals
compRegister(c, "fleet", compFiles)
compRegister(c, "env", compEnvs)
return c
}

// runFleetDashboard opens the view. The terminal check comes first — before
// the fleet file is even read — so a piped invocation fails the same way
// wherever the fleet file stands: it never half-enters the view.
Expand Down
2 changes: 1 addition & 1 deletion cmd/spinloop/fleet_dashboard_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2205,7 +2205,7 @@ func TestDashProgramStartsAndStopsANode(t *testing.T) {
// the command that carries the same data into a pipe.
func TestFleetDashboardRefusesPipedOutput(t *testing.T) {
captureStdout(t, func() {
err := cmdFleet([]string{"dashboard"})
err := cmdDashboard(nil)
if err == nil {
t.Fatal("dashboard ran without a terminal")
}
Expand Down
18 changes: 10 additions & 8 deletions cmd/spinloop/fleet_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ func twoNodeFleet(t *testing.T, state string) *httptest.Server {
func TestCmdFleetStatusRendersEveryNode(t *testing.T) {
twoNodeFleet(t, "running")
out := captureStdout(t, func() {
if err := cmdFleet([]string{"status"}); err != nil {
if err := cmdStatus(nil); err != nil {
// One unreachable node must not fail the command.
t.Errorf("fleet status returned %v", err)
}
Expand All @@ -133,7 +133,7 @@ func TestCmdFleetStatusShowsVersion(t *testing.T) {
"nodes:\n - name: node\n host: %s\n port: %d\n",
host, port))
out := captureStdout(t, func() {
if err := cmdFleet([]string{"status"}); err != nil {
if err := cmdStatus(nil); err != nil {
t.Error(err)
}
})
Expand Down Expand Up @@ -628,7 +628,7 @@ func TestCmdFleetUnknownNodeNamesTheKnownOnes(t *testing.T) {

func TestCmdFleetMissingFileNamesThePath(t *testing.T) {
t.Chdir(t.TempDir())
err := cmdFleet([]string{"status"})
err := cmdStatus(nil)
if err == nil {
t.Fatal("missing fleet file accepted")
}
Expand All @@ -649,7 +649,7 @@ func TestCmdFleetExplicitPath(t *testing.T) {
// Somewhere else entirely, so only --fleet can find it.
t.Chdir(t.TempDir())
out := captureStdout(t, func() {
if err := cmdFleet([]string{"status", "--fleet", path}); err != nil {
if err := cmdStatus([]string{"--fleet", path}); err != nil {
t.Error(err)
}
})
Expand All @@ -672,7 +672,7 @@ func TestCmdFleetExplicitPathShortForm(t *testing.T) {
// Somewhere else entirely, so only -f can find it.
t.Chdir(t.TempDir())
out := captureStdout(t, func() {
if err := cmdFleet([]string{"status", "-f", path}); err != nil {
if err := cmdStatus([]string{"-f", path}); err != nil {
t.Error(err)
}
})
Expand All @@ -687,7 +687,9 @@ func TestFleetFlagShortForm(t *testing.T) {
isolateConfig(t)
root := newRootCmd()
fleet := commandUnder(t, root, "fleet")
for _, name := range []string{"status", "metrics", "start", "stop", "deploy", "route", "dashboard"} {
// status and dashboard are top-level verbs now; the rest still hang off
// the group, and every one of them offers -f for the fleet file.
for _, name := range []string{"metrics", "start", "stop", "deploy", "route"} {
sub := commandUnder(t, fleet, name)
f := sub.Flags().Lookup("fleet")
if f == nil {
Expand Down Expand Up @@ -786,7 +788,7 @@ func TestCmdFleetStatusShowsIdleTime(t *testing.T) {
writeFleetFile(t, fmt.Sprintf("nodes:\n - name: busy\n host: %s\n port: %d\n", host, port))

out := captureStdout(t, func() {
if err := cmdFleet([]string{"status"}); err != nil {
if err := cmdStatus(nil); err != nil {
t.Error(err)
}
})
Expand All @@ -810,7 +812,7 @@ func TestCmdFleetStatusOmitsIdleWithoutActivity(t *testing.T) {
writeFleetFile(t, fmt.Sprintf("nodes:\n - name: fresh\n host: %s\n port: %d\n", host, port))

out := captureStdout(t, func() {
if err := cmdFleet([]string{"status"}); err != nil {
if err := cmdStatus(nil); err != nil {
t.Error(err)
}
})
Expand Down
58 changes: 58 additions & 0 deletions cmd/spinloop/status.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
// `spinloop status`: what every engine in the target is doing, one row each.
// The target is whatever names one — a registered environment, a fleet file,
// or the fleet file in the working directory — so an environment and a fleet
// holding that same environment read identically. The rendering lives beside
// the fleet's other views in fleet.go; this is the command.

package main

import (
"context"
"os"

"github.com/spf13/cobra"
"github.com/spinloop-ai/spinloop/internal/fleet"
)

func statusCmd() *cobra.Command {
var path, envName string
c := &cobra.Command{
Use: "status",
Short: "report every engine's state",
Long: `reports what each engine in the target is doing: its state, what it
serves, how long since it last did work, and the spinloop version of the
daemon running it. One row per node, queried concurrently, so the command
takes as long as the slowest node rather than all of them added up.

The target is a registered environment (--env), a fleet file (--fleet), or
the fleet.yaml in the working directory. A node that cannot be reached is a
row saying so, not a failure: one unreachable machine never blanks the rest.

An environment's endpoint address is spinloop remote env, and its retention
deadline spinloop fleet metrics — neither is a column here, because neither
applies to every node.`,
Args: cobra.NoArgs,
SilenceErrors: true,
SilenceUsage: true,
RunE: func(c *cobra.Command, _ []string) error {
resolve(c)
cfg, err := resolveFleetTarget(fleetTarget{envName: envName, fleetPath: path})
if err != nil {
return err
}
results := cfg.FanOut(context.Background(), fleet.StatusCall)
renderFleetStatus(os.Stdout, results)
return nil
},
}
fs := c.Flags()
fs.StringVarP(&path, "fleet", "f", "", fleetFileUsage)
fs.StringVar(&envName, "env", "", envFlagTargetUsage)
c.ValidArgsFunction = noPositionals
compRegister(c, "fleet", compFiles)
compRegister(c, "env", compEnvs)
return c
}

// cmdStatus runs the command through the tree — the seam the suite calls.
func cmdStatus(args []string) error { return execCmd(statusCmd(), args) }
Loading