From 0420fbd01d84d2f98c5ce3c02b53dbd932f86464 Mon Sep 17 00:00:00 2001 From: Jason McCartney Date: Tue, 8 Sep 2026 16:15:31 -0700 Subject: [PATCH] Keep SparkGLM standalone and make LLooM integration opt-in SparkGLM's default installer should run independently. Use Docker and SSH for source builds, pinned model downloads, worker-first startup and owned-container lifecycle. Keep the existing LLooM-managed flow behind --lloom; it requires an existing LLooM installation and never installs the gateway. Both paths retain the measured NVFP4 settings, with latest EXL3 explicitly selectable. Provenance: - Copy the measured standalone-capable entrypoint byte-for-byte from https://github.com/Enntity/lloom at 2cc2f0df9ddcb1bb7fe60f7bd6934d2a9de1e4f2, preserving its MIT AND Apache-2.0 notice and the repository license ledger. This script invokes vLLM directly and contains no LLooM runtime dependency. Original work: - Standalone Docker/SSH orchestration, private-fabric binding, owned-container checks, source/default dispatch, optional integration documentation and command-parity tests. No numerical engine changes or new performance claims. Verification: - scripts/check.sh all passed, including measured per-rank argument parity, rejection of foreign container ownership, standalone default and optional LLooM dispatch. All 66 checksum-bound result bundles remain valid. GPU runtime was not restarted; the existing LLooM-managed appliance remains in service. --- README.md | 2 +- SPARKGLM.md | 148 +++++++++----------- docs/ATTRIBUTION.md | 6 + docs/LLOOM_INSTALL.md | 106 +++++++++++++++ docs/X_NVFP4_UPDATE_DRAFT.md | 1 + provenance/upstreams.json | 12 +- runtime/entrypoint.sh | 166 +++++++++++++++++++++++ scripts/appliance.py | 2 +- scripts/check.sh | 1 + scripts/check_commit_provenance.py | 2 + scripts/standalone.py | 211 +++++++++++++++++++++++++++++ start.sh | 8 +- tests/test_standalone.py | 51 +++++++ 13 files changed, 628 insertions(+), 88 deletions(-) create mode 100644 docs/LLOOM_INSTALL.md create mode 100755 runtime/entrypoint.sh create mode 100644 scripts/standalone.py create mode 100644 tests/test_standalone.py diff --git a/README.md b/README.md index 2667477..a91ca6f 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ SparkGLM targets responsive **concurrent GLM-5.3-Flash serving on two NVIDIA DGX Spark GB10 systems**, building on MiaAI-Lab's excellent two-Spark work. `main` now defaults to our measured **NVFP4** path: native CUTLASS W4A4, MXFP8 DFlash2, 512K context, 9 GiB KV per rank, 2K chunks and mixed scheduling. -LLooM controls the locally built runtime. The latest EXL3 work remains on the +SparkGLM runs independently with Docker and SSH; LLooM integration is optional. The latest EXL3 work remains on the [`exl3` branch](https://github.com/Enntity/sparkglm/tree/exl3). > **Source research preview:** this is the maintainer-selected measured default, diff --git a/SPARKGLM.md b/SPARKGLM.md index e40bad6..8794043 100644 --- a/SPARKGLM.md +++ b/SPARKGLM.md @@ -1,104 +1,88 @@ -# Run SparkGLM: NVFP4 on two DGX Sparks +# Run SparkGLM independently on two DGX Sparks -The default on `main` is the maintainer-selected **NVFP4 research preview**: -512K context (524,288 tokens), 9 GiB KV per rank, native FlashInfer CUTLASS, -MXFP8 DFlash2 TP2 with seven draft tokens, 2K prefill chunks, four active -sequences, and mixed scheduling. This is the configuration recorded in the -September 8 mixed video. `--skip` reproduces the other video setting. +SparkGLM installs and serves directly with **Docker and SSH**. It does not +install or require LLooM. NVFP4 is the default: 524288-token context, 9 GiB KV +per rank, native FlashInfer CUTLASS, MXFP8 DFlash2 TP2 k7, 2K chunks and mixed +scheduling. `--skip` selects the recorded NVFP4 comparison option; +`--profile exl3` selects the latest concurrent E3 EXL3 profile with 1M context. -This is a source install. No prebuilt image or complete G5 endurance/general -quality certification is claimed. See [results](results/CURRENT.md), -[limitations](docs/KNOWN_LIMITATIONS.md), and [model licenses](docs/LICENSING.md). +This is a source research preview, not a G5-certified appliance. Read +[the results](results/CURRENT.md) and [model licenses](docs/LICENSING.md). -## Prepare the cluster +## Prerequisites -Use two Linux ARM64 DGX Sparks with working NVIDIA Container Toolkit, Docker, -Python 3 with venv, Git, rsync, and passwordless SSH from leader to worker. -Use the same user/home and LLooM install path on both nodes. Configure LLooM's -[direct two-node cluster](https://github.com/Enntity/lloom/blob/main/docs/clusters.md) -and verify its fabric addresses, interface, and RoCE mapping before installing. -This launcher uses that existing cluster configuration; it does not invent -network settings or change application aliases. +On both Linux ARM64 DGX Sparks: Docker with NVIDIA Container Toolkit, Python 3 +with venv, Git, rsync, and passwordless SSH from leader to worker. Use matching +account home paths and the same absolute model-root path. Configure the direct +TP fabric and inspect `ip address`, `ibdev2netdev` and the populated RoCE v2 GID +tables to obtain each node's interface, RDMA device and GID index. -Install LLooM from source on **both** nodes at the revision in -[`profiles/build.json`](profiles/build.json), using its documented `npm ci` -and `npm link` steps. For example, on each node: +Stop other resident full models through their current manager before building +or installing. Native compilation requires at least 32 GiB MemAvailable on the +leader. Allow roughly 200 GB for the target plus the draft, source/build layers, +images and caches on each node; 190 GB total free space is not sufficient. +The API binds to the supplied leader fabric address; keep it on your trusted +private network or put an authenticated gateway in front of it. -```bash -git clone https://github.com/Enntity/lloom.git -cd lloom -git checkout 2cc2f0df9ddcb1bb7fe60f7bd6934d2a9de1e4f2 -npm ci -npm link -``` - -Configure and run its gateway service following the LLooM instructions. The launcher checks the SparkGLM entrypoint hash on both -nodes. LLooM owns admission, worker-first startup, readiness, routing and stop. - -Allow at least 32 GiB **MemAvailable** on the leader for native compilation: -stop resident full models through LLooM first. Reserve ample disk for roughly -200 GB of target weights, the draft, Docker source/build layers, and the -second local weight copy on the worker; 190 GB is not enough for a fresh build. -Weights remain separately downloaded, pinned publisher artifacts. +## Install -## Install and start - -Run on the Spark leader: +Run on the leader. Replace the uppercase placeholders with your configuration: ```bash git clone https://github.com/Enntity/sparkglm.git cd sparkglm ./start.sh plan -./start.sh --worker USER@WORKER +./start.sh --worker USER@WORKER \ + --head-address HEAD_FABRIC_IP --worker-address WORKER_FABRIC_IP \ + --interface HEAD_INTERFACE --worker-interface WORKER_INTERFACE \ + --hca HEAD_RDMA_DEVICE --worker-hca WORKER_RDMA_DEVICE \ + --gid HEAD_GID_INDEX --worker-gid WORKER_GID_INDEX ``` -Replace `USER@WORKER` with the SSH destination. `--model-root /path/to/models` -selects the same absolute directory on both nodes; the default is -`~/.lloom/models`. `--lloom-root /path/to/lloom` overrides executable-based -installation discovery. - -The installer builds the pinned source layers in order, copies the immutable -image to the worker, downloads pinned models through LLooM, copies those local -weights to the worker, installs an additive managed recipe, and starts it. -Existing models/default aliases remain registered. Reinstalling stops only -the selected SparkGLM runtime before replacing its configuration; clients using it will be interrupted. -A successful install ends with LLooM runtime status. Call the configured LLooM -gateway using model **`sparkglm-nvfp4`** and your gateway authentication. - -For an image you already built and qualified, avoid rebuilding with: +The worker interface, HCA and GID default to the leader's values when their +worker overrides are omitted. The installer builds the pinned source layers, +copies the immutable image, downloads pinned weights into +`~/.cache/sparkglm/models`, copies the weights locally to the worker, and starts +worker then leader. `--model-root /path/to/models` reuses your existing weight +location; `--image sha256:FULL_IMAGE_ID` reuses a qualified local build. +No daemon or package manager beyond Docker is required for serving. + +The default endpoint is `http://HEAD_FABRIC_IP:8890/v1`, model +`sparkglm-nvfp4`. A successful start checks the exact model identity. Its +standalone containers are named `sparkglm-standalone-nvfp4-head` and +`sparkglm-standalone-nvfp4-worker`; use `docker logs` on their respective nodes +for startup diagnostics. The default readiness budget is two hours because a +first source build/model load is not a quick prebuilt-image installation. ```bash -./start.sh --worker USER@WORKER --image sha256:YOUR_FULL_IMAGE_ID +./start.sh status +./start.sh stop +./start.sh start ``` -Both rank identities are checked. For separately built and qualified rank images, add `--worker-image sha256:WORKER_IMAGE_ID`. Use the `check` command with these image arguments to validate the installed adapter, image identities and LLooM setup plan without downloading weights or changing runtime configuration. A different image is your own experiment, -not automatically the recorded source. To reproduce skip scheduling, add -`--skip` to the install command. It retains our existing 3584-token remaining -prefill bypass and zero max-wait setting. Changing scheduling requires reinstall; -`start` simply starts the already installed profile. +Lifecycle commands read the saved local standalone installation. They operate +only on containers bearing SparkGLM's standalone ownership label. There is no +independent auto-restart policy; an unsuccessful launch stops its ranks. Keep +the same worker and profile when restarting. For EXL3, include `--profile exl3` +in each command. The old EXL3 video-foundation launcher remains separately +available as `start-exl3.sh`; see [its historical guide](docs/EXL3_QUICKSTART.md). + +Build pins are in [profiles/build.json](profiles/build.json) and +[profiles/build-exl3.json](profiles/build-exl3.json). The +[public source map](provenance/2026-09-08-publication-map.json) connects them to +the measured original source trees. Rebuilding does not promise bit-identical +images; hardware qualification remains necessary for a new build. + +## Optional LLooM integration + +If you already use LLooM, it can own the same SparkGLM runtime instead: ```bash -./start.sh status -./start.sh stop -./start.sh start +./start.sh --lloom --worker USER@WORKER +./start.sh --lloom status ``` -The build pins retain the measured EXL3 foundation and intermediate adapter -layers, then add the measured NVFP4/MXFP8 support. E3 remains inactive for -NVFP4. Public source snapshots preserve original tree bytes; the -[revision map](provenance/2026-09-08-publication-map.json) connects them to -historical measurement SHAs. Rebuilding is not a claim of bit-identical images. - -## EXL3 and research - -The latest EXL3 work is preserved on the -[`exl3` branch](https://github.com/Enntity/sparkglm/tree/exl3), including the -corrected concurrent E3 policy, 32-row threshold, and 1M profile. -Use `./start.sh --profile exl3 --worker USER@WORKER` from `main` to build and install that latest EXL3 profile through LLooM. It has its own runtime identity; stop the active full model first. The old standalone launcher is retained -as `start-exl3.sh`; it reproduces the older video foundation, **not** the latest -E3 profile. Its historical instructions are in -[EXL3_QUICKSTART.md](docs/EXL3_QUICKSTART.md). - -For experiments, use [METHODOLOGY.md](docs/METHODOLOGY.md): model-free operator -checks, tinyGLM integration, matched full-model workloads, and semantic checks. -Do not equate synthetic fixture success with model quality or capacity. +See [LLooM integration](docs/LLOOM_INSTALL.md) for its cluster prerequisites and +immutable-image recipe. SparkGLM does not install LLooM even in this mode. +Choose one lifecycle owner; stop the active standalone or managed runtime +before switching. This changes management, not the measured inference recipe. diff --git a/docs/ATTRIBUTION.md b/docs/ATTRIBUTION.md index 0d282cb..7840458 100644 --- a/docs/ATTRIBUTION.md +++ b/docs/ATTRIBUTION.md @@ -142,3 +142,9 @@ https://github.com/Enntity/lloom at The MIT notice is retained in `LICENSES/MIT-LLooM.txt`. The installer is original Apache-2.0 orchestration and calls LLooM's existing commands. Its installed entrypoint is hash checked against the measured version, not copied here. + +The standalone `runtime/entrypoint.sh` is a byte-for-byte copy of that measured +MIT/Apache launcher (also published in LLooM at +`2cc2f0df9ddcb1bb7fe60f7bd6934d2a9de1e4f2`). It executes the selected image's +vLLM and patchers directly. It does not invoke, install, or depend on LLooM. +The standalone Docker/SSH orchestrator is original Apache-2.0 code. diff --git a/docs/LLOOM_INSTALL.md b/docs/LLOOM_INSTALL.md new file mode 100644 index 0000000..c31568f --- /dev/null +++ b/docs/LLOOM_INSTALL.md @@ -0,0 +1,106 @@ +# Optional LLooM integration for SparkGLM + +This opt-in integration requires an existing LLooM installation. SparkGLM itself is independent; see [standalone installation](../SPARKGLM.md). + +The selected profile is the maintainer-selected **NVFP4 research preview**: +512K context (524,288 tokens), 9 GiB KV per rank, native FlashInfer CUTLASS, +MXFP8 DFlash2 TP2 with seven draft tokens, 2K prefill chunks, four active +sequences, and mixed scheduling. This is the configuration recorded in the +September 8 mixed video. `--skip` reproduces the other video setting. + +This is a source install. No prebuilt image or complete G5 endurance/general +quality certification is claimed. See [results](../results/CURRENT.md), +[limitations](KNOWN_LIMITATIONS.md), and [model licenses](LICENSING.md). + +## Prepare the cluster + +Use two Linux ARM64 DGX Sparks with working NVIDIA Container Toolkit, Docker, +Python 3 with venv, Git, rsync, and passwordless SSH from leader to worker. +Use the same user/home and LLooM install path on both nodes. Configure LLooM's +[direct two-node cluster](https://github.com/Enntity/lloom/blob/main/docs/clusters.md) +and verify its fabric addresses, interface, and RoCE mapping before installing. +This launcher uses that existing cluster configuration; it does not invent +network settings or change application aliases. + +Install LLooM from source on **both** nodes at the revision in +[`profiles/build.json`](../profiles/build.json), using its documented `npm ci` +and `npm link` steps. For example, on each node: + +```bash +git clone https://github.com/Enntity/lloom.git +cd lloom +git checkout 2cc2f0df9ddcb1bb7fe60f7bd6934d2a9de1e4f2 +npm ci +npm link +``` + +Configure and run its gateway service following the LLooM instructions. The launcher checks the SparkGLM entrypoint hash on both +nodes. LLooM owns admission, worker-first startup, readiness, routing and stop. + +Allow at least 32 GiB **MemAvailable** on the leader for native compilation: +stop resident full models through LLooM first. Reserve ample disk for roughly +200 GB of target weights, the draft, Docker source/build layers, and the +second local weight copy on the worker; 190 GB is not enough for a fresh build. +Weights remain separately downloaded, pinned publisher artifacts. + +## Install and start + +Run on the Spark leader: + +```bash +git clone https://github.com/Enntity/sparkglm.git +cd sparkglm +./start.sh --lloom plan +./start.sh --lloom --worker USER@WORKER +``` + +Replace `USER@WORKER` with the SSH destination. `--model-root /path/to/models` +selects the same absolute directory on both nodes; the default is +`~/.lloom/models`. `--lloom-root /path/to/lloom` overrides executable-based +installation discovery. + +The installer builds the pinned source layers in order, copies the immutable +image to the worker, downloads pinned models through LLooM, copies those local +weights to the worker, installs an additive managed recipe, and starts it. +Existing models/default aliases remain registered. Reinstalling stops only +the selected SparkGLM runtime before replacing its configuration; clients using it will be interrupted. +A successful install ends with LLooM runtime status. Call the configured LLooM +gateway using model **`sparkglm-nvfp4`** and your gateway authentication. + +For an image you already built and qualified, avoid rebuilding with: + +```bash +./start.sh --lloom --worker USER@WORKER --image sha256:YOUR_FULL_IMAGE_ID +``` + +Both rank identities are checked. For separately built and qualified rank images, add `--worker-image sha256:WORKER_IMAGE_ID`. Use the `check` command with these image arguments to validate the installed adapter, image identities and LLooM setup plan without downloading weights or changing runtime configuration. A different image is your own experiment, +not automatically the recorded source. To reproduce skip scheduling, add +`--skip` to the install command. It retains our existing 3584-token remaining +prefill bypass and zero max-wait setting. Changing scheduling requires reinstall; +`start` simply starts the already installed profile. + +```bash +./start.sh --lloom status +./start.sh --lloom stop +./start.sh --lloom start +``` + +The build pins retain the measured EXL3 foundation and intermediate adapter +layers, then add the measured NVFP4/MXFP8 support. E3 remains inactive for +NVFP4. Public source snapshots preserve original tree bytes; the +[revision map](../provenance/2026-09-08-publication-map.json) connects them to +historical measurement SHAs. Rebuilding is not a claim of bit-identical images. + +## EXL3 and research + +The latest EXL3 work is preserved on the +[`exl3` branch](https://github.com/Enntity/sparkglm/tree/exl3), including the +corrected concurrent E3 policy, 32-row threshold, and 1M profile. +Use `./start.sh --lloom --profile exl3 --worker USER@WORKER` from `main` to build and install that latest EXL3 profile through LLooM. It has its own runtime identity; stop the active full model first. The old standalone launcher is retained +as `start-exl3.sh`; it reproduces the older video foundation, **not** the latest +E3 profile. Its historical instructions are in +[EXL3_QUICKSTART.md](EXL3_QUICKSTART.md). + +For experiments, use [METHODOLOGY.md](METHODOLOGY.md): model-free operator +checks, tinyGLM integration, matched full-model workloads, and semantic checks. +Do not equate synthetic fixture success with model quality or capacity. diff --git a/docs/X_NVFP4_UPDATE_DRAFT.md b/docs/X_NVFP4_UPDATE_DRAFT.md index bb7438b..7956fb4 100644 --- a/docs/X_NVFP4_UPDATE_DRAFT.md +++ b/docs/X_NVFP4_UPDATE_DRAFT.md @@ -39,6 +39,7 @@ staggered C4 workload: ours mixed vs Mia's default skip, then skip vs skip. Both show SparkGLM on top. Context, drafts and settings are disclosed. Latest NVFP4 is on main; latest EXL3 stays available on the exl3 branch. +SparkGLM runs independently; LLooM integration is optional. Source, exact settings and results: https://github.com/Enntity/sparkglm EXL3 checkpoint credit: Brandon M. Music's ShapleyMCG (2026), “ShapleyMCG: diff --git a/provenance/upstreams.json b/provenance/upstreams.json index bd68ce2..85f7a40 100644 --- a/provenance/upstreams.json +++ b/provenance/upstreams.json @@ -260,10 +260,11 @@ "id": "lloom-managed-recipe", "repository": "https://github.com/Enntity/lloom", "revision": "75e08ca0923e7eb58f087b53032c82f88175ea3a", - "relationship": "Adapted SparkGLM managed recipe metadata with the measured September 8 settings; entrypoint remains supplied by LLooM and hash checked.", + "relationship": "Adapted SparkGLM managed recipe metadata with the measured September 8 settings; entrypoint remains supplied by LLooM and hash checked. Standalone entrypoint copied byte-for-byte from the same measured adapter; it executes vLLM directly and has no LLooM dependency.", "covered_paths": [ "profiles/nvfp4.json", - "profiles/exl3.json" + "profiles/exl3.json", + "runtime/entrypoint.sh" ], "notice_ids": [ "MIT-LLooM" @@ -271,6 +272,13 @@ } ], "path_rules": [ + { + "patterns": [ + "runtime/entrypoint.sh" + ], + "license_expression": "MIT AND Apache-2.0", + "reason": "Measured launcher copied from LLooM, retaining inherited Mia and SparkGLM notices" + }, { "patterns": [ "profiles/nvfp4.json", diff --git a/runtime/entrypoint.sh b/runtime/entrypoint.sh new file mode 100755 index 0000000..a100ae5 --- /dev/null +++ b/runtime/entrypoint.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MIT AND Apache-2.0 +# Adapted from LLooM's Mia launcher; SparkGLM supplies the built runtime. +set -euo pipefail + +log() { printf '[sparkglm rank=%s] %s\n' "${NODE_RANK:-?}" "$*"; } + +: "${NODE_RANK:?NODE_RANK is required}" +: "${CLUSTER_NODE_COUNT:?CLUSTER_NODE_COUNT is required}" +: "${MASTER_ADDR:?MASTER_ADDR is required}" +: "${MODEL_DIR:?MODEL_DIR is required}" + +# The inherited DFlash loader preserves the target parallel configuration; +# accepting draft TP1 here would silently run TP2 and mislabel measurements. +if [[ "${SPEC_METHOD:-dflash}" == "dflash" && "${DFLASH_DRAFT_TP:-2}" != "${CLUSTER_NODE_COUNT}" ]]; then + log "independent DFlash draft TP is not implemented by this adapter; use target TP=${CLUSTER_NODE_COUNT}" + exit 1 +fi + +[[ -f "${MODEL_DIR}/config.json" ]] || { + log "missing target config: ${MODEL_DIR}/config.json" + exit 1 +} + +if [[ "${SPARKGLM_EXL3_E3:-0}" == "1" ]]; then + [[ -f /usr/local/lib/python3.12/dist-packages/sparkglm_e3.py && -f /usr/local/lib/python3.12/dist-packages/exl3_fat_moe_ext.so ]] || { + log "selected image does not contain the E3 adapter and extension"; exit 1; + } + if [[ "${SPARKGLM_EXL3_E3_POLICY:-large}" == "concurrent" ]]; then + [[ -f /usr/local/lib/python3.12/dist-packages/sparkglm_e3_policy.py ]] || { + log "selected image does not contain the concurrent E3 policy"; exit 1; + } + fi +fi +if [[ "${SPARKGLM_NVFP4_TINY:-0}" == "1" ]]; then + [[ -f /usr/local/lib/python3.12/dist-packages/sparkglm_nvfp4_tiny.py && -f /usr/local/lib/python3.12/dist-packages/sparkglm_nvfp4_tiny.pth ]] || { + log "selected image does not contain the guarded NVFP4 fixture initializer"; exit 1; + } +fi +if [[ "${SPARKGLM_MXFP8_DRAFT:-0}" == "1" ]]; then + grep -q '_fused_kv_weight_scale' /opt/glm53/patch_dflash2.py || { + log "selected image does not contain MXFP8 DFlash2 context projection support"; exit 1; + } +fi + +if [[ "${SPEC_METHOD:-dflash}" == "dflash" && ! -f "${DFLASH_MODEL_DIR:-}/config.json" ]]; then + log "missing DFlash2 config: ${DFLASH_MODEL_DIR:-unset}/config.json" + exit 1 +fi + +# Runtime patches come from the selected SparkGLM image, never the legacy +# Mia files installed alongside LLooM. Image identity is pinned by the recipe. +for patch in \ + patch_glm_video_placeholders.py \ + patch_suppress_stops_in_reasoning.py \ + patch_scheduler_decode_floor.py \ + patch_glm5_drafter_group.py \ + patch_hybrid_prefix_hit.py \ + patch_xgrammar_termination.py \ + patch_kpool_tail_slotmap.py \ + patch_spinwait.py \ + patch_indexer_workspace.py \ + patch_ablit.py; do + [[ -f "/opt/glm53/${patch}" ]] || { + log "missing SparkGLM runtime patch: /opt/glm53/${patch}" + exit 1 + } + # These source patchers use only the standard library. Avoid importing the + # serving stack through site .pth hooks ten times during each cold start. + python3 -S "/opt/glm53/${patch}" +done + +if [[ -z "${LIMIT_MM_PER_PROMPT:-}" ]]; then + LIMIT_MM_PER_PROMPT='{"image":4,"video":1}' +fi + +args=( + --served-model-name "${SERVED_MODEL_NAME:-glm-5.3-flash-exl3}" + --host "${VLLM_HOST:-0.0.0.0}" + --port "${VLLM_PORT:-8890}" + --tensor-parallel-size "${CLUSTER_NODE_COUNT}" + --nnodes "${CLUSTER_NODE_COUNT}" + --node-rank "${NODE_RANK}" + --master-addr "${MASTER_ADDR}" + --master-port "${MASTER_PORT:-29521}" + --distributed-executor-backend mp + --tool-call-parser glm47 + --enable-auto-tool-choice + --reasoning-parser glm45 + --enable-prefix-caching + --no-enable-flashinfer-autotune + --quantization "${QUANTIZATION:-exl3}" + --max-model-len "${MAX_MODEL_LEN:-1000000}" + --gpu-memory-utilization "${GPU_MEMORY_UTILIZATION:-0.87}" + --max-num-seqs "${MAX_NUM_SEQS:-4}" + --max-num-batched-tokens "${MAX_NUM_BATCHED_TOKENS:-7168}" + --kv-cache-dtype "${KV_CACHE_DTYPE:-fp8}" + --chat-template /opt/glm53/chat_template.jinja + --limit-mm-per-prompt "${LIMIT_MM_PER_PROMPT}" + --skip-mm-profiling +) + +if [[ "${SPARKGLM_TINY_DUMMY:-0}" == "1" ]]; then + python3 -S - "${MODEL_DIR}/config.json" <<'PYSAFE' +import json, sys +c = json.load(open(sys.argv[1])) +if c.get("quantization_config", {}).get("version") != "tinyglm-v1": + raise SystemExit("dummy loading requires the synthetic tinyGLM fixture") +PYSAFE + [[ "${SPEC_METHOD:-dflash}" == "none" ]] || { log "tinyGLM requires SPEC_METHOD=none"; exit 1; } + args+=(--load-format dummy --generation-config vllm) +fi +if [[ "${SPARKGLM_NVFP4_TINY:-0}" == "1" ]]; then + python3 -S - "${MODEL_DIR}/config.json" <<'PYSAFE' +import json, sys +if json.load(open(sys.argv[1])).get("_sparkglm_fixture") != "tinyglm-nvfp4-v1": + raise SystemExit("NVFP4 dummy loading requires the synthetic fixture") +PYSAFE + [[ "${SPEC_METHOD:-dflash}" == "none" && "${QUANTIZATION:-exl3}" == "compressed-tensors" ]] || { log "invalid NVFP4 fixture options"; exit 1; } + args+=(--load-format dummy --generation-config vllm) +fi +if [[ "${LANGUAGE_MODEL_ONLY:-0}" == "1" ]]; then + args+=(--language-model-only) +fi + +if [[ -n "${KV_CACHE_MEMORY_BYTES:-}" ]]; then + args+=(--kv-cache-memory-bytes "${KV_CACHE_MEMORY_BYTES}") +fi +if [[ -n "${MOE_BACKEND:-}" ]]; then + args+=(--moe-backend "${MOE_BACKEND}") +fi + +if [[ "${NODE_RANK}" != "0" ]]; then + args+=(--headless) +fi + +case "${SPEC_METHOD:-dflash}" in + dflash) + dflash_tokens="${DFLASH_TOKENS:-7}" + dflash_draft_tp="${DFLASH_DRAFT_TP:-2}" + [[ "${dflash_tokens}" =~ ^[0-9]+$ ]] || { log "invalid DFLASH_TOKENS=${dflash_tokens}"; exit 1; } + [[ "${dflash_draft_tp}" =~ ^[0-9]+$ ]] || { log "invalid DFLASH_DRAFT_TP=${dflash_draft_tp}"; exit 1; } + # Do not launch Python here: Mia's installed video .pth emits a status line + # on interpreter startup, which would contaminate command-substitution JSON. + printf -v spec '{"method":"dflash","model":"%s","num_speculative_tokens":%d,"kv_cache_dtype":"auto","draft_sample_method":"probabilistic","rejection_sample_method":"standard","draft_tensor_parallel_size":%d}' \ + "${DFLASH_MODEL_DIR}" "${dflash_tokens}" "${dflash_draft_tp}" + args+=(--speculative-config "${spec}") + ;; + mtp) + args+=(--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":${MTP_TOKENS:-2}}") + ;; + none) ;; + *) + log "unsupported SPEC_METHOD=${SPEC_METHOD}" + exit 1 + ;; +esac + +if [[ "${ENFORCE_EAGER:-0}" == "1" ]]; then + args+=(--enforce-eager) +else + args+=(--cudagraph-capture-sizes 1 2 4 8 16 24 32) +fi + +log "starting SparkGLM quant=${QUANTIZATION:-exl3} TP=${CLUSTER_NODE_COUNT}, spec=${SPEC_METHOD:-dflash}, draft-tp=${DFLASH_DRAFT_TP:-2}, mnbt=${MAX_NUM_BATCHED_TOKENS:-7168}, E2=${EXL3_FAT_KERNEL:-1}" +exec vllm serve "${MODEL_DIR}" "${args[@]}" diff --git a/scripts/appliance.py b/scripts/appliance.py index 6f9c81b..9689d44 100644 --- a/scripts/appliance.py +++ b/scripts/appliance.py @@ -34,7 +34,7 @@ def build(cache: Path, profile: str = 'nvfp4') -> str: raise RuntimeError('Build on the Spark leader (Linux ARM64), not a Mac or x86 machine.') available = next(int(x.split()[1]) for x in Path('/proc/meminfo').read_text().splitlines() if x.startswith('MemAvailable:')) if available < 32 * 1024**2: - raise RuntimeError('Native compilation needs 32 GiB MemAvailable. Stop resident models through LLooM before building.') + raise RuntimeError('Native compilation needs 32 GiB MemAvailable. Stop resident models through their runtime manager before building.') manifest = json.loads((ROOT / ('profiles/build-exl3.json' if profile == 'exl3' else 'profiles/build.json')).read_text()) previous = None for layer in manifest['layers']: diff --git a/scripts/check.sh b/scripts/check.sh index 054c4ec..f6ab0fe 100755 --- a/scripts/check.sh +++ b/scripts/check.sh @@ -6,6 +6,7 @@ ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." >/dev/null 2>&1 && pwd)" cd "$ROOT" QUICK_TESTS=( + tests/test_standalone.py tests/test_appliance.py tests/test_bringup_robustness.py tests/test_benchmark_contract.py diff --git a/scripts/check_commit_provenance.py b/scripts/check_commit_provenance.py index 1433bf5..3e717a6 100755 --- a/scripts/check_commit_provenance.py +++ b/scripts/check_commit_provenance.py @@ -17,6 +17,8 @@ "start.sh", "start-exl3.sh", "scripts/appliance.py", + "scripts/standalone.py", + "runtime/", "profiles/", "scripts/boot-shape-warmup.sh", ) diff --git a/scripts/standalone.py b/scripts/standalone.py new file mode 100644 index 0000000..8cd5eb1 --- /dev/null +++ b/scripts/standalone.py @@ -0,0 +1,211 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Install and serve SparkGLM directly with Docker and SSH; no gateway required.""" +from __future__ import annotations + +import argparse +import ipaddress +import json +from pathlib import Path +import re +import subprocess +import time +import urllib.error +import urllib.request + +from appliance import ROOT, build, materialize, remote, run + +OWNER = 'org.enntity.sparkglm.owner=standalone' + + +def command(config, rank, *args, capture=False): + if rank: + return remote(config['worker'], *args, capture=capture) + return run(*args, capture=capture) + + +def containers(config, action): + """Operate only containers bearing our ownership label, never LLooM's.""" + for rank in (0, 1): + name = config['names'][rank] + ids = command(config, rank, 'docker', 'ps', '-aq', '--filter', 'name=^/' + name + '$', capture=True) + if not ids: + continue + owned = command(config, rank, 'docker', 'ps', '-aq', '--filter', 'name=^/' + name + '$', '--filter', 'label=' + OWNER, capture=True) + if ids != owned: + raise RuntimeError('Refusing to operate on a container not owned by this standalone installer: ' + name) + if action == 'stop': + command(config, rank, 'docker', 'stop', '--time', '30', name) + elif action == 'remove': + command(config, rank, 'docker', 'rm', '-f', name) + elif action == 'status': + print(command(config, rank, 'docker', 'inspect', '--format', '{{.Name}} {{.State.Status}} {{.Image}}', name, capture=True)) + + +def docker_args(recipe, config, rank): + member = recipe['models'][0]['settings']['placement']['members'][1 if rank == 0 else 0] + source = member['runtimeSettings']['bootstrap']['createArgs'] + address = config['head_address'] if rank == 0 else config['worker_address'] + substitutions = {'modelRoot':config['model_root'], 'nodeAddress':address, + 'fabricInterface':config['interfaces'][rank], 'nodeRank':str(rank), + 'clusterNodeCount':'2', 'leaderAddress':config['head_address']} + result = ['docker', 'run', '-d', '--name', config['names'][rank], '--label', OWNER] + for value in source: + if value.startswith('type=bind,src=${lloomRoot}'): + value = 'type=bind,src=' + config['entrypoint'] + ',dst=/opt/sparkglm/entrypoint.sh,readonly' + for key, replacement in substitutions.items(): + value = value.replace('${' + key + '}', replacement) + if '${' in value: + raise RuntimeError('Unresolved runtime parameter: ' + value) + value = value.replace('lloom-glm53-', 'sparkglm-standalone-') + if value.startswith('VLLM_HOST='): + value = 'VLLM_HOST=' + address + result.append(value) + result += ['-e', 'VLLM_HOST_IP=' + address, + '-e', 'NCCL_IB_HCA=' + config['hcas'][rank], + '-e', 'NCCL_IB_GID_INDEX=' + str(config['gids'][rank]), + config['images'][rank], '/opt/sparkglm/entrypoint.sh'] + return result + + +def start(config, timeout): + recipe = materialize(config['images'][0], config['skip'], config['profile']) + # Only replace this installer's named containers. A failed start cleans up both ranks. + containers(config, 'remove') + try: + command(config, 1, *docker_args(recipe, config, 1)) + command(config, 0, *docker_args(recipe, config, 0)) + port = recipe['models'][0]['settings']['port'] + model = recipe['models'][0]['gatewayModel'] + url = f"http://{config['head_address']}:{port}/v1/models" + opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + for rank in (0, 1): + running = command(config, rank, 'docker', 'inspect', '--format', '{{.State.Running}}', config['names'][rank], capture=True) + if running != 'true': + raise RuntimeError('Rank exited; inspect docker logs for ' + config['names'][rank]) + try: + with opener.open(url, timeout=5) as response: + models = json.load(response).get('data', []) + if any(item.get('id') == model for item in models): + print(f'Ready: {url.removesuffix("/models")} model={model}') + return + except (urllib.error.URLError, TimeoutError, ValueError): + pass + time.sleep(5) + raise RuntimeError('Model readiness timed out; inspect the rank logs before retrying.') + except BaseException: + containers(config, 'stop') + raise + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('command', nargs='?', default='install', choices=['plan','build','install','start','stop','status']) + parser.add_argument('--profile', choices=['nvfp4','exl3'], default='nvfp4') + parser.add_argument('--worker', help='SSH destination USER@WORKER') + parser.add_argument('--head-address', help='Leader IPv4 address on the private TP fabric') + parser.add_argument('--worker-address', help='Worker IPv4 address on the private TP fabric') + parser.add_argument('--interface', help='Leader fabric interface') + parser.add_argument('--worker-interface', help='Worker interface; defaults to --interface') + parser.add_argument('--hca', help='Leader RDMA device from ibdev2netdev') + parser.add_argument('--worker-hca', help='Worker RDMA device; defaults to --hca') + parser.add_argument('--gid', type=int, help='Leader populated RoCE v2 GID index') + parser.add_argument('--worker-gid', type=int, help='Worker GID index; defaults to --gid') + parser.add_argument('--image', help='Reuse a qualified immutable local image ID') + parser.add_argument('--skip', action='store_true', help='Recorded NVFP4 skip comparison option') + parser.add_argument('--model-root', type=Path, default=Path.home()/'.cache/sparkglm/models') + parser.add_argument('--ready-timeout', type=int, default=7200) + args = parser.parse_args() + if args.skip and args.profile == 'exl3': + parser.error('--skip is the recorded NVFP4 option') + if args.ready_timeout <= 0: + parser.error('--ready-timeout must be positive') + if args.command == 'plan': + recipe = materialize('sha256:'+'0'*64, args.skip, args.profile) + manifest = 'build-exl3.json' if args.profile == 'exl3' else 'build.json' + print(json.dumps({'manager':'standalone', 'profile':recipe, + 'build':json.loads((ROOT/'profiles'/manifest).read_text()), + 'scheduling':'skip' if args.skip else 'mixed'}, indent=2)) + return + cache = Path.home()/'.cache/sparkglm' + cache.mkdir(parents=True, exist_ok=True) + state = cache/('standalone-'+args.profile+'.json') + if args.command in ('start','stop','status'): + if not state.exists(): + parser.error('No standalone installation exists for this profile; run install first.') + config = json.loads(state.read_text()) + if args.command == 'start': + start(config, args.ready_timeout) + else: + containers(config, args.command) + return + if args.command == 'build': + print(build(cache, args.profile)) + return + for flag in ('worker','head_address','worker_address','interface','hca','gid'): + if getattr(args, flag) is None: + parser.error('--'+flag.replace('_','-')+' is required; see SPARKGLM.md') + if args.worker.startswith('-'): + parser.error('Invalid worker SSH destination') + for address in (args.head_address, args.worker_address): + ipaddress.IPv4Address(address) + for value in (args.interface,args.worker_interface or args.interface,args.hca,args.worker_hca or args.hca): + if not re.fullmatch(r'[A-Za-z0-9_.:-]+',value): + parser.error('Invalid fabric interface or RDMA device') + gids = [args.gid, args.worker_gid if args.worker_gid is not None else args.gid] + if any(x < 0 for x in gids): + parser.error('GID indices must be nonnegative') + model_root = str(args.model_root.expanduser().resolve()) + if not re.fullmatch(r'/[A-Za-z0-9_./-]+', model_root): + parser.error('Use a model root without spaces or shell metacharacters') + # Matching paths simplify the read-only two-rank mounts; no gateway is needed. + if remote(args.worker, 'printenv', 'HOME', capture=True) != str(Path.home()): + parser.error('Use matching account home paths on the two Sparks') + run('docker','version') + remote(args.worker,'docker','version') + image = args.image or build(cache,args.profile) + recipe = materialize(image,args.skip,args.profile) + if run('docker','image','inspect','--format','{{.Id}}',image,capture=True) != image: + raise RuntimeError('Local image identity mismatch') + try: + worker_image = remote(args.worker,'docker','image','inspect','--format','{{.Id}}',image,capture=True) + except subprocess.CalledProcessError: + worker_image = '' + if worker_image != image: + save = subprocess.Popen(['docker','save',image],stdout=subprocess.PIPE) + load = subprocess.run(['ssh','-o','BatchMode=yes',args.worker,'docker load'],stdin=save.stdout) + save.stdout.close() + if save.wait() or load.returncode: + raise RuntimeError('Image transfer failed') + if remote(args.worker,'docker','image','inspect','--format','{{.Id}}',image,capture=True) != image: + raise RuntimeError('Worker image identity mismatch') + env = cache/'hf-cli' + if not (env/'bin/hf').exists(): + run('python3','-m','venv',str(env)) + run(str(env/'bin/pip'),'install','huggingface_hub[cli]==0.35.3') + for step in recipe['setup']['steps']: + if step.get('action') == 'download-model': + directory = str(Path(model_root)/step['model'].replace('/','--')) + run(str(env/'bin/hf'),'download',step['model'],'--revision',step['revision'],'--local-dir',directory) + remote(args.worker,'mkdir','-p',directory) + run('rsync','-a','--partial',directory+'/',args.worker+':'+directory+'/') + entrypoint = cache/'runtime-entrypoint.sh' + entrypoint.write_bytes((ROOT/'runtime/entrypoint.sh').read_bytes()) + remote(args.worker,'mkdir','-p',str(cache)) + run('scp',str(entrypoint),args.worker+':'+str(entrypoint)) + config = dict(profile=args.profile,worker=args.worker,head_address=args.head_address, + worker_address=args.worker_address,model_root=model_root,entrypoint=str(entrypoint), + interfaces=[args.interface,args.worker_interface or args.interface], + hcas=[args.hca,args.worker_hca or args.hca],gids=gids,images=[image,image],skip=args.skip, + names=['sparkglm-standalone-'+args.profile+'-head','sparkglm-standalone-'+args.profile+'-worker']) + state.write_text(json.dumps(config,indent=2)+'\n') + start(config,args.ready_timeout) + + +if __name__ == '__main__': + try: + main() + except (RuntimeError, ValueError, subprocess.CalledProcessError) as error: + raise SystemExit(str(error)) from error diff --git a/start.sh b/start.sh index 954e9c3..586caba 100755 --- a/start.sh +++ b/start.sh @@ -1,6 +1,10 @@ #!/usr/bin/env bash # SPDX-License-Identifier: Apache-2.0 -# Default: measured NVFP4 profile under LLooM. EXL3 remains explicitly selectable. +# Standalone NVFP4 by default. LLooM integration is explicitly optional. set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -exec python3 "$ROOT/scripts/appliance.py" "$@" +if [[ "${1:-}" == "--lloom" ]]; then + shift + exec python3 "$ROOT/scripts/appliance.py" "$@" +fi +exec python3 "$ROOT/scripts/standalone.py" "$@" diff --git a/tests/test_standalone.py b/tests/test_standalone.py new file mode 100644 index 0000000..8bbd3b9 --- /dev/null +++ b/tests/test_standalone.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Protect standalone ownership and measured serving arguments without a gateway.""" +import json +from pathlib import Path +import subprocess +import sys +from unittest.mock import patch + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0,str(ROOT/'scripts')) +import standalone +from appliance import materialize + +image='sha256:'+'a'*64 +config=dict(worker='worker',head_address='192.0.2.1',worker_address='192.0.2.2', + model_root='/models',entrypoint='/runtime/entrypoint.sh',interfaces=['eth0','eth1'], + hcas=['rdma0','rdma1'],gids=[3,4],images=[image,image],names=['test-head','test-worker']) +for profile in ('nvfp4','exl3'): + recipe=materialize(image,profile=profile) + for rank in (0,1): + args=standalone.docker_args(recipe,config,rank) + assert not any('lloom' in x or '${' in x for x in args), args + assert args[-2:]==[image,'/opt/sparkglm/entrypoint.sh'] + assert standalone.OWNER in args + assert 'NODE_RANK='+str(rank) in args + assert 'NCCL_IB_GID_INDEX='+str(config['gids'][rank]) in args + assert 'VLLM_HOST='+config['head_address' if rank==0 else 'worker_address'] in args + for member in recipe['models'][0]['settings']['placement']['members']: + for value in member['runtimeSettings']['bootstrap']['createArgs']: + if value.startswith(('MAX_MODEL_LEN=','KV_CACHE_MEMORY_BYTES=','MAX_NUM_BATCHED_TOKENS=', + 'DFLASH_','MOE_BACKEND=','GLM53_MIXED_PREFILL_CHUNK=')): + assert value in args,value +calls=[] +def foreign(config,rank,*args,capture=False): + calls.append(args) + return '' if 'label='+standalone.OWNER in args else 'foreign-container' +with patch.object(standalone,'command',foreign): + try: + standalone.containers(config,'remove') + raise AssertionError('foreign container accepted') + except RuntimeError: + pass +assert all('rm' not in x and 'stop' not in x for x in calls) +plan=json.loads(subprocess.check_output([str(ROOT/'start.sh'),'plan'],text=True)) +assert plan['manager']=='standalone' +assert plan['profile']['models'][0]['gatewayModel']=='sparkglm-nvfp4' +optional=json.loads(subprocess.check_output([str(ROOT/'start.sh'),'--lloom','plan'],text=True)) +assert optional['profile']['models'][0]['gatewayModel']=='sparkglm-nvfp4' +assert (ROOT/'runtime/entrypoint.sh').read_bytes().startswith(b'#!/usr/bin/env bash') +print('Standalone command parity, ownership isolation and optional LLooM dispatch PASS')