Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ SparkGLM targets responsive **concurrent GLM-5.3-Flash serving on two NVIDIA
DGX Spark GB10 systems**, building on MiaAI-Lab's excellent two-Spark work.
`main` now defaults to our measured **NVFP4** path: native CUTLASS W4A4,
MXFP8 DFlash2, 512K context, 9 GiB KV per rank, 2K chunks and mixed scheduling.
LLooM controls the locally built runtime. The latest EXL3 work remains on the
SparkGLM runs independently with Docker and SSH; LLooM integration is optional. The latest EXL3 work remains on the
[`exl3` branch](https://github.com/Enntity/sparkglm/tree/exl3).

> **Source research preview:** this is the maintainer-selected measured default,
Expand Down
148 changes: 66 additions & 82 deletions SPARKGLM.md
Original file line number Diff line number Diff line change
@@ -1,104 +1,88 @@
# Run SparkGLM: NVFP4 on two DGX Sparks
# Run SparkGLM independently on two DGX Sparks

The default on `main` is the maintainer-selected **NVFP4 research preview**:
512K context (524,288 tokens), 9 GiB KV per rank, native FlashInfer CUTLASS,
MXFP8 DFlash2 TP2 with seven draft tokens, 2K prefill chunks, four active
sequences, and mixed scheduling. This is the configuration recorded in the
September 8 mixed video. `--skip` reproduces the other video setting.
SparkGLM installs and serves directly with **Docker and SSH**. It does not
install or require LLooM. NVFP4 is the default: 524288-token context, 9 GiB KV
per rank, native FlashInfer CUTLASS, MXFP8 DFlash2 TP2 k7, 2K chunks and mixed
scheduling. `--skip` selects the recorded NVFP4 comparison option;
`--profile exl3` selects the latest concurrent E3 EXL3 profile with 1M context.

This is a source install. No prebuilt image or complete G5 endurance/general
quality certification is claimed. See [results](results/CURRENT.md),
[limitations](docs/KNOWN_LIMITATIONS.md), and [model licenses](docs/LICENSING.md).
This is a source research preview, not a G5-certified appliance. Read
[the results](results/CURRENT.md) and [model licenses](docs/LICENSING.md).

## Prepare the cluster
## Prerequisites

Use two Linux ARM64 DGX Sparks with working NVIDIA Container Toolkit, Docker,
Python 3 with venv, Git, rsync, and passwordless SSH from leader to worker.
Use the same user/home and LLooM install path on both nodes. Configure LLooM's
[direct two-node cluster](https://github.com/Enntity/lloom/blob/main/docs/clusters.md)
and verify its fabric addresses, interface, and RoCE mapping before installing.
This launcher uses that existing cluster configuration; it does not invent
network settings or change application aliases.
On both Linux ARM64 DGX Sparks: Docker with NVIDIA Container Toolkit, Python 3
with venv, Git, rsync, and passwordless SSH from leader to worker. Use matching
account home paths and the same absolute model-root path. Configure the direct
TP fabric and inspect `ip address`, `ibdev2netdev` and the populated RoCE v2 GID
tables to obtain each node's interface, RDMA device and GID index.

Install LLooM from source on **both** nodes at the revision in
[`profiles/build.json`](profiles/build.json), using its documented `npm ci`
and `npm link` steps. For example, on each node:
Stop other resident full models through their current manager before building
or installing. Native compilation requires at least 32 GiB MemAvailable on the
leader. Allow roughly 200 GB for the target plus the draft, source/build layers,
images and caches on each node; 190 GB total free space is not sufficient.
The API binds to the supplied leader fabric address; keep it on your trusted
private network or put an authenticated gateway in front of it.

```bash
git clone https://github.com/Enntity/lloom.git
cd lloom
git checkout 2cc2f0df9ddcb1bb7fe60f7bd6934d2a9de1e4f2
npm ci
npm link
```

Configure and run its gateway service following the LLooM instructions. The launcher checks the SparkGLM entrypoint hash on both
nodes. LLooM owns admission, worker-first startup, readiness, routing and stop.

Allow at least 32 GiB **MemAvailable** on the leader for native compilation:
stop resident full models through LLooM first. Reserve ample disk for roughly
200 GB of target weights, the draft, Docker source/build layers, and the
second local weight copy on the worker; 190 GB is not enough for a fresh build.
Weights remain separately downloaded, pinned publisher artifacts.
## Install

## Install and start

Run on the Spark leader:
Run on the leader. Replace the uppercase placeholders with your configuration:

```bash
git clone https://github.com/Enntity/sparkglm.git
cd sparkglm
./start.sh plan
./start.sh --worker USER@WORKER
./start.sh --worker USER@WORKER \
--head-address HEAD_FABRIC_IP --worker-address WORKER_FABRIC_IP \
--interface HEAD_INTERFACE --worker-interface WORKER_INTERFACE \
--hca HEAD_RDMA_DEVICE --worker-hca WORKER_RDMA_DEVICE \
--gid HEAD_GID_INDEX --worker-gid WORKER_GID_INDEX
```

Replace `USER@WORKER` with the SSH destination. `--model-root /path/to/models`
selects the same absolute directory on both nodes; the default is
`~/.lloom/models`. `--lloom-root /path/to/lloom` overrides executable-based
installation discovery.

The installer builds the pinned source layers in order, copies the immutable
image to the worker, downloads pinned models through LLooM, copies those local
weights to the worker, installs an additive managed recipe, and starts it.
Existing models/default aliases remain registered. Reinstalling stops only
the selected SparkGLM runtime before replacing its configuration; clients using it will be interrupted.
A successful install ends with LLooM runtime status. Call the configured LLooM
gateway using model **`sparkglm-nvfp4`** and your gateway authentication.

For an image you already built and qualified, avoid rebuilding with:
The worker interface, HCA and GID default to the leader's values when their
worker overrides are omitted. The installer builds the pinned source layers,
copies the immutable image, downloads pinned weights into
`~/.cache/sparkglm/models`, copies the weights locally to the worker, and starts
worker then leader. `--model-root /path/to/models` reuses your existing weight
location; `--image sha256:FULL_IMAGE_ID` reuses a qualified local build.
No daemon or package manager beyond Docker is required for serving.

The default endpoint is `http://HEAD_FABRIC_IP:8890/v1`, model
`sparkglm-nvfp4`. A successful start checks the exact model identity. Its
standalone containers are named `sparkglm-standalone-nvfp4-head` and
`sparkglm-standalone-nvfp4-worker`; use `docker logs` on their respective nodes
for startup diagnostics. The default readiness budget is two hours because a
first source build/model load is not a quick prebuilt-image installation.

```bash
./start.sh --worker USER@WORKER --image sha256:YOUR_FULL_IMAGE_ID
./start.sh status
./start.sh stop
./start.sh start
```

Both rank identities are checked. For separately built and qualified rank images, add `--worker-image sha256:WORKER_IMAGE_ID`. Use the `check` command with these image arguments to validate the installed adapter, image identities and LLooM setup plan without downloading weights or changing runtime configuration. A different image is your own experiment,
not automatically the recorded source. To reproduce skip scheduling, add
`--skip` to the install command. It retains our existing 3584-token remaining
prefill bypass and zero max-wait setting. Changing scheduling requires reinstall;
`start` simply starts the already installed profile.
Lifecycle commands read the saved local standalone installation. They operate
only on containers bearing SparkGLM's standalone ownership label. There is no
independent auto-restart policy; an unsuccessful launch stops its ranks. Keep
the same worker and profile when restarting. For EXL3, include `--profile exl3`
in each command. The old EXL3 video-foundation launcher remains separately
available as `start-exl3.sh`; see [its historical guide](docs/EXL3_QUICKSTART.md).

Build pins are in [profiles/build.json](profiles/build.json) and
[profiles/build-exl3.json](profiles/build-exl3.json). The
[public source map](provenance/2026-09-08-publication-map.json) connects them to
the measured original source trees. Rebuilding does not promise bit-identical
images; hardware qualification remains necessary for a new build.

## Optional LLooM integration

If you already use LLooM, it can own the same SparkGLM runtime instead:

```bash
./start.sh status
./start.sh stop
./start.sh start
./start.sh --lloom --worker USER@WORKER
./start.sh --lloom status
```

The build pins retain the measured EXL3 foundation and intermediate adapter
layers, then add the measured NVFP4/MXFP8 support. E3 remains inactive for
NVFP4. Public source snapshots preserve original tree bytes; the
[revision map](provenance/2026-09-08-publication-map.json) connects them to
historical measurement SHAs. Rebuilding is not a claim of bit-identical images.

## EXL3 and research

The latest EXL3 work is preserved on the
[`exl3` branch](https://github.com/Enntity/sparkglm/tree/exl3), including the
corrected concurrent E3 policy, 32-row threshold, and 1M profile.
Use `./start.sh --profile exl3 --worker USER@WORKER` from `main` to build and install that latest EXL3 profile through LLooM. It has its own runtime identity; stop the active full model first. The old standalone launcher is retained
as `start-exl3.sh`; it reproduces the older video foundation, **not** the latest
E3 profile. Its historical instructions are in
[EXL3_QUICKSTART.md](docs/EXL3_QUICKSTART.md).

For experiments, use [METHODOLOGY.md](docs/METHODOLOGY.md): model-free operator
checks, tinyGLM integration, matched full-model workloads, and semantic checks.
Do not equate synthetic fixture success with model quality or capacity.
See [LLooM integration](docs/LLOOM_INSTALL.md) for its cluster prerequisites and
immutable-image recipe. SparkGLM does not install LLooM even in this mode.
Choose one lifecycle owner; stop the active standalone or managed runtime
before switching. This changes management, not the measured inference recipe.
6 changes: 6 additions & 0 deletions docs/ATTRIBUTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,3 +142,9 @@ https://github.com/Enntity/lloom at
The MIT notice is retained in `LICENSES/MIT-LLooM.txt`. The installer is original
Apache-2.0 orchestration and calls LLooM's existing commands. Its installed
entrypoint is hash checked against the measured version, not copied here.

The standalone `runtime/entrypoint.sh` is a byte-for-byte copy of that measured
MIT/Apache launcher (also published in LLooM at
`2cc2f0df9ddcb1bb7fe60f7bd6934d2a9de1e4f2`). It executes the selected image's
vLLM and patchers directly. It does not invoke, install, or depend on LLooM.
The standalone Docker/SSH orchestrator is original Apache-2.0 code.
106 changes: 106 additions & 0 deletions docs/LLOOM_INSTALL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Optional LLooM integration for SparkGLM

This opt-in integration requires an existing LLooM installation. SparkGLM itself is independent; see [standalone installation](../SPARKGLM.md).

The selected profile is the maintainer-selected **NVFP4 research preview**:
512K context (524,288 tokens), 9 GiB KV per rank, native FlashInfer CUTLASS,
MXFP8 DFlash2 TP2 with seven draft tokens, 2K prefill chunks, four active
sequences, and mixed scheduling. This is the configuration recorded in the
September 8 mixed video. `--skip` reproduces the other video setting.

This is a source install. No prebuilt image or complete G5 endurance/general
quality certification is claimed. See [results](../results/CURRENT.md),
[limitations](KNOWN_LIMITATIONS.md), and [model licenses](LICENSING.md).

## Prepare the cluster

Use two Linux ARM64 DGX Sparks with working NVIDIA Container Toolkit, Docker,
Python 3 with venv, Git, rsync, and passwordless SSH from leader to worker.
Use the same user/home and LLooM install path on both nodes. Configure LLooM's
[direct two-node cluster](https://github.com/Enntity/lloom/blob/main/docs/clusters.md)
and verify its fabric addresses, interface, and RoCE mapping before installing.
This launcher uses that existing cluster configuration; it does not invent
network settings or change application aliases.

Install LLooM from source on **both** nodes at the revision in
[`profiles/build.json`](../profiles/build.json), using its documented `npm ci`
and `npm link` steps. For example, on each node:

```bash
git clone https://github.com/Enntity/lloom.git
cd lloom
git checkout 2cc2f0df9ddcb1bb7fe60f7bd6934d2a9de1e4f2
npm ci
npm link
```

Configure and run its gateway service following the LLooM instructions. The launcher checks the SparkGLM entrypoint hash on both
nodes. LLooM owns admission, worker-first startup, readiness, routing and stop.

Allow at least 32 GiB **MemAvailable** on the leader for native compilation:
stop resident full models through LLooM first. Reserve ample disk for roughly
200 GB of target weights, the draft, Docker source/build layers, and the
second local weight copy on the worker; 190 GB is not enough for a fresh build.
Weights remain separately downloaded, pinned publisher artifacts.

## Install and start

Run on the Spark leader:

```bash
git clone https://github.com/Enntity/sparkglm.git
cd sparkglm
./start.sh --lloom plan
./start.sh --lloom --worker USER@WORKER
```

Replace `USER@WORKER` with the SSH destination. `--model-root /path/to/models`
selects the same absolute directory on both nodes; the default is
`~/.lloom/models`. `--lloom-root /path/to/lloom` overrides executable-based
installation discovery.

The installer builds the pinned source layers in order, copies the immutable
image to the worker, downloads pinned models through LLooM, copies those local
weights to the worker, installs an additive managed recipe, and starts it.
Existing models/default aliases remain registered. Reinstalling stops only
the selected SparkGLM runtime before replacing its configuration; clients using it will be interrupted.
A successful install ends with LLooM runtime status. Call the configured LLooM
gateway using model **`sparkglm-nvfp4`** and your gateway authentication.

For an image you already built and qualified, avoid rebuilding with:

```bash
./start.sh --lloom --worker USER@WORKER --image sha256:YOUR_FULL_IMAGE_ID
```

Both rank identities are checked. For separately built and qualified rank images, add `--worker-image sha256:WORKER_IMAGE_ID`. Use the `check` command with these image arguments to validate the installed adapter, image identities and LLooM setup plan without downloading weights or changing runtime configuration. A different image is your own experiment,
not automatically the recorded source. To reproduce skip scheduling, add
`--skip` to the install command. It retains our existing 3584-token remaining
prefill bypass and zero max-wait setting. Changing scheduling requires reinstall;
`start` simply starts the already installed profile.

```bash
./start.sh --lloom status
./start.sh --lloom stop
./start.sh --lloom start
```

The build pins retain the measured EXL3 foundation and intermediate adapter
layers, then add the measured NVFP4/MXFP8 support. E3 remains inactive for
NVFP4. Public source snapshots preserve original tree bytes; the
[revision map](../provenance/2026-09-08-publication-map.json) connects them to
historical measurement SHAs. Rebuilding is not a claim of bit-identical images.

## EXL3 and research

The latest EXL3 work is preserved on the
[`exl3` branch](https://github.com/Enntity/sparkglm/tree/exl3), including the
corrected concurrent E3 policy, 32-row threshold, and 1M profile.
Use `./start.sh --lloom --profile exl3 --worker USER@WORKER` from `main` to build and install that latest EXL3 profile through LLooM. It has its own runtime identity; stop the active full model first. The old standalone launcher is retained
as `start-exl3.sh`; it reproduces the older video foundation, **not** the latest
E3 profile. Its historical instructions are in
[EXL3_QUICKSTART.md](EXL3_QUICKSTART.md).

For experiments, use [METHODOLOGY.md](METHODOLOGY.md): model-free operator
checks, tinyGLM integration, matched full-model workloads, and semantic checks.
Do not equate synthetic fixture success with model quality or capacity.
1 change: 1 addition & 0 deletions docs/X_NVFP4_UPDATE_DRAFT.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ staggered C4 workload: ours mixed vs Mia's default skip, then skip vs skip.
Both show SparkGLM on top. Context, drafts and settings are disclosed.

Latest NVFP4 is on main; latest EXL3 stays available on the exl3 branch.
SparkGLM runs independently; LLooM integration is optional.
Source, exact settings and results: https://github.com/Enntity/sparkglm

EXL3 checkpoint credit: Brandon M. Music's ShapleyMCG (2026), “ShapleyMCG:
Expand Down
12 changes: 10 additions & 2 deletions provenance/upstreams.json
Original file line number Diff line number Diff line change
Expand Up @@ -260,17 +260,25 @@
"id": "lloom-managed-recipe",
"repository": "https://github.com/Enntity/lloom",
"revision": "75e08ca0923e7eb58f087b53032c82f88175ea3a",
"relationship": "Adapted SparkGLM managed recipe metadata with the measured September 8 settings; entrypoint remains supplied by LLooM and hash checked.",
"relationship": "Adapted SparkGLM managed recipe metadata with the measured September 8 settings; entrypoint remains supplied by LLooM and hash checked. Standalone entrypoint copied byte-for-byte from the same measured adapter; it executes vLLM directly and has no LLooM dependency.",
"covered_paths": [
"profiles/nvfp4.json",
"profiles/exl3.json"
"profiles/exl3.json",
"runtime/entrypoint.sh"
],
"notice_ids": [
"MIT-LLooM"
]
}
],
"path_rules": [
{
"patterns": [
"runtime/entrypoint.sh"
],
"license_expression": "MIT AND Apache-2.0",
"reason": "Measured launcher copied from LLooM, retaining inherited Mia and SparkGLM notices"
},
{
"patterns": [
"profiles/nvfp4.json",
Expand Down
Loading