Validate CUDA F16 compressed attention KV on DGX Spark

[AustinKelsay]

Parent

Parent PRD: #1

What to build

Run the DGX Spark validation and benchmark ladder for the experimental CUDA F16
compressed attention KV build. The end-to-end behavior is an evidence packet
that compares the experimental build against the current DS4 frontdoor
checkpoint and says clearly whether the result is correct enough and fast enough
to keep pursuing.

This slice requires DGX Spark access and may remain HITL until the implementation
commit is ready to test on the Spark host.

Acceptance criteria

• Experimental build passes make cuda-regression on DGX Spark.
• Experimental build passes a short CLI smoke on DGX Spark.
• Experimental build passes at least one long-context prompt on DGX Spark.
• Direct before/after ds4-bench results are captured for 32K, 128K, and one stress context when stable.
• Benchmark evidence records prefill, decode, memory, and managed-KV fallback logs.
• A distinct experimental F16 frontdoor alias is used before frontdoor checks.
• Existing frontdoor speed probe and smoke checks are rerun through spark-cluster.
• Toolcall-15 is rerun if direct and frontdoor checks pass.
• HermesAgent-20 is rerun only after cheaper checks pass.

Blocked by

• Preserve default CUDA behavior while isolating experimental F16 dispatch #5

[AustinKelsay]

Partial Spark validation checkpoint recorded in commit 0ba1d0f and .scratch/feature-dev/cuda-f16-compressed-kv/spark-validation-2026-06-22.md.

Passed on spark-123a in a separate validation clone at /home/finite/ds4-cuda-f16-compressed-kv:

• Default CUDA build: make clean; make cuda-spark
• Experimental CUDA build: make clean; make cuda-spark DS4_CUDA_ATTN_COMP_CACHE_F16=1
• CUDA regression: make cuda-regression
• Regression result: cuda long-context regression: OK

Not yet run:

• Experimental short CLI smoke
• Experimental long-context prompt
• Direct ds4-bench before/after comparison
• Frontdoor smoke through a distinct experimental F16 alias
• Toolcall-15 and HermesAgent-20

Reason for keeping this issue open: spark-123a is actively serving DS4 on port 8000 and the process holds about 105 GB of GPU memory. The other checked Sparks were also occupied, so I left all live runtimes untouched. Remaining validation needs a free Spark runtime slot or explicit approval to move/stop an existing runtime.

[AustinKelsay]

Spark validation now has live-slot benchmark evidence from spark-123a.

What changed operationally:

• Operator approval was given to temporarily replace the live DS4 server on spark-123a.
• Experimental clone: /home/finite/ds4-cuda-f16-compressed-kv
• Commit tested: 0ba1d0f
• Build: make cuda-spark DS4_CUDA_ATTN_COMP_CACHE_F16=1
• Experimental server ran on :8000, then the original baseline DS4 server was restored.
• Restored baseline sanity passed through the operator frontdoor.

Evidence:

• DS4 branch checkpoint commit: 379f7ae
• Feature checkpoint: .scratch/feature-dev/cuda-f16-compressed-kv/spark-f16-benchmark-2026-06-22.md
• Spark-cluster run: runs/2026-06-22-ds4-cuda-f16-kv-spark123a-experiment/README.md

Results versus the current DS4 baseline:

• Decode 256 cap: 15.025 tok/s vs baseline 15.205 tok/s (-1.2%)
• Long prefill wall: 147.953 prompt tok/s vs baseline 342.585 prompt tok/s (-56.8%)
• Operator frontdoor smoke: passed
• Toolcall-15: score 93, 14 pass / 1 fail, same score as baseline; wall 208.416s vs 200.74s
• HermesAgent-20: score 73.25, 10 pass / 2 partial / 8 fail vs baseline 73.75, 10 pass / 3 partial / 7 fail; wall 1440.474s vs 1476.206s

Interpretation: the prototype is functionally viable but not performance viable. The likely cause is that F16 compressed-KV reads currently fall through the generic CUDA attention path instead of the optimized F32 heads/static kernels. I am leaving this issue open rather than closing it because the original checklist included direct ds4-bench and a distinct experimental alias; we intentionally used the live alias for the operator-approved in-place experiment and got enough evidence to reject this first prototype before spending more time on direct bench variants.

Recommended next slice: implement an optimized CUDA F16 compressed-KV attention path, then rerun this validation ladder.

[AustinKelsay]

Optimized F16 compressed-KV heads8 validation is complete and pushed in 41cd5e9 on feature/cuda-f16-compressed-kv.

Evidence checkpoint: /Users/plebdev/spark-cluster/runs/2026-06-22-ds4-cuda-f16-heads8-kv-spark123a-optimized/README.md

Key results versus the existing DS4 baseline and the first F16 prototype:

Probe	Baseline	Initial F16	Optimized F16 heads8
Decode cap	15.205 tok/s	15.025 tok/s	15.284 tok/s
Long prefill wall	342.585 prompt tok/s	147.953 prompt tok/s	338.439 prompt tok/s
Toolcall-15	93; 14 pass, 1 fail	93; 14 pass, 1 fail	93; 14 pass, 1 fail
HermesAgent-20	73.75; 10 pass, 3 partial, 7 fail	73.25; 10 pass, 2 partial, 8 fail	73.75; 11 pass, 2 partial, 7 fail

Validation run covered default and F16 CUDA builds, default and F16 make cuda-regression with cuda long-context regression: OK, live operator frontdoor smoke, speed probe, Toolcall-15, and HermesAgent-20. The temporary F16 runtime was run on spark-123a with operator approval, then the baseline DS4 server was restored and frontdoor sanity passed with ds4-restored-ready.

Interpretation: the first F16 prototype was functionally viable but too slow; the heads8 source-load optimization recovered the main 124k long-prefill regression, moving from -56.8% versus baseline to about -1.2% versus baseline. This remains opt-in/experimental and should still be trimmed before any upstream PR.

[AustinKelsay]

Final checkpoint for the optimized F16 compressed-KV heads8 slice is pushed.

Commits:

• DS4 fork: 66b7bff on feature/cuda-f16-compressed-kv adds the optimization next-steps checkpoint.
• Spark checkpoint: 455bde2e on finitecomputer/spark-cluster updates the run README with next optimization targets.

New DS4 planning artifact:
.scratch/feature-dev/cuda-f16-compressed-kv/optimization-next-steps-2026-06-22.md

Run artifact with mirrored next targets:
/Users/plebdev/spark-cluster/runs/2026-06-22-ds4-cuda-f16-heads8-kv-spark123a-optimized/README.md

Recommendation: the next meaningful optimization slice is not another front-door behavior run. It is a direct DS4 benchmark ladder at 32k, 128k, 256k/512k, and a stress context if stable, with GPU memory and KV disk usage captured around each run. If F16 is going to become a true win, it should show up as compressed-cache size grows or as reduced memory pressure/fallback behavior.