Help/pr 132 - #400
Draft
samuelishida wants to merge 15 commits into
Draft
Conversation
What: port selected portable behavior from PRs FlashML-org#132, FlashML-org#131, FlashML-org#241, FlashML-org#137, FlashML-org#135, FlashML-org#217, and FlashML-org#378 into the upstream base, including HIP toolchain support, RCCL guards, GGUF metadata/readers, Qwen3.5 GGUF loading, native Q4/K-quant dense and MoE paths, sampler penalties, and graph-safe CPU/hybrid execution. Why: give PR FlashML-org#132 a reviewable ROCm/GGUF foundation without importing gfx1100-only defaults or silently claiming unsupported targets. Unknown architectures, unsafe graph capture, missing native candidates, incomplete benchmark evidence, and stale cache identity now fail closed or remain explicit fallback states. How: select HIP/CUDA flags centrally; use tracked HIP headers and exact RDNA target capabilities; preserve raw-versus-slot expert ID contracts; keep candidate kernels opt-in; add weighted route reduction, replay/provenance manifests, decode gates, ROCm profiling, backend/toolchain/source-derived JIT keys, and scoped stale-lock cleanup. Validation: consolidated ROCm run passed 225 tests with 4 skips; fresh isolated TVM-FFI JIT passed 7 tests; live gfx1100 Q4_0, Q5_K, and Q6_K native parity passed. Environment used shared /home/smk/.venv/vllm-rocm, Torch 2.10.0+git8514f05, ROCm 7.2.1, Triton 3.6.0. No CUDA nvcc, non-gfx1100 hardware, real Qwen GGUF serving fixture, repeated A/B promotion run, or rocprof trace was available, so no cross-arch serving or performance promotion is claimed.
What: add indexed institutional notes for ROCm/GGUF consolidation decisions and remaining evidence gaps. Why: keep future plan and implementation cycles from re-litigating portability, opt-in promotion, and JIT-cache requirements. How: document hardest decision, rejected alternatives, least-confident serving claim, and reuse paths under .agents/learnings/.
What: preserve per-tensor Qwen GGUF quant types, add native Q4_K/Q5_K/Q6_K/Q8_0 expert dispatch, convert CPU expert sources to Q4_0, and provide an in-tree ROCm LRU fallback.\n\nWhy: real Qwen3.5 Q4_K_S exports mixed expert rows, while the ROCm environment lacks CUDA-only flashlib. Fixed Q4_K/Q5_K assumptions and the first-decode offload import failure without creating another large environment.\n\nHow: scan GGUF headers for layer/type metadata, carry exact types and padded strides through resident/offload GEMV, convert CPU sources one tensor at a time, select Triton full-fetch LRU on HIP, and normalize rocprofv3 traces.\n\nValidation: 100 targeted tests passed; Python compileall passed; real gfx1100 Qwen3.5 CPU and GPU-offload one-token HTTP 200 finite-output smoke passed. Backend outputs were intentionally not treated as parity evidence.
What: update ROCm target, install, model, and consolidation-plan records with Qwen3.5 GGUF CPU and native GPU smoke evidence.\n\nWhy: implementation status changed from fixture-pending to verified gfx1100 serving, while other targets, graph replay, native/reference parity, and performance promotion remain unproven.\n\nHow: record the fixture SHA256, backend-specific one-token results, mixed quant coverage, Triton LRU fallback, and exact compile-only versus served target boundaries.\n\nValidation: documentation diff checked with git diff --check; evidence matches the 100-test gate and completed gfx1100 server runs.
What: record the mixed-quant Qwen GGUF and flashlib-free ROCm decisions in indexed repository learnings.\n\nWhy: future plan and implementation cycles must not reintroduce Q4_K-only assumptions, CUDA-only cache dependencies, or unsupported parity claims.\n\nHow: document exact source-native GPU versus Q4_0 CPU contracts, rejected alternatives, and remaining confidence boundaries in the existing consolidation topic.\n\nValidation: learning index and Markdown structure checked with git diff --check.
What: add Qwen35moeGGUFForCausalLM as an alias of the existing Qwen3.5 AOT geometry entry.\n\nWhy: the new GGUF registry key made the established registry-coverage test reject an otherwise intentional model variant.\n\nHow: declare the architecture alias while keeping mixed native GGUF experts on their GGUF JIT path, separate from safetensors bank formats.\n\nValidation: targeted AOT registry and Qwen MoE tests passed 9 tests.
What: document the registry-to-AOT alias rule uncovered by full-suite validation.\n\nWhy: GGUF-only architecture keys share model geometry but not safetensors bank formats, so future changes need explicit coverage without false AOT claims.\n\nHow: use AotModel.arch_aliases for shared geometry and keep native mixed GGUF dispatch on its JIT path.\n\nValidation: AOT registry and Qwen MoE tests passed 9 tests.
What: update plan claims for Qwen3.5 registry coverage and mixed native GGUF dispatch.\n\nWhy: prior evidence still described Gemma-only registration, omitted Q4_K/Q5_K, and called mixed merged rows unsupported after implementation landed.\n\nHow: align source references and evidence with current registry, layer, and merged-linear behavior.\n\nValidation: plan diff passed git diff --check; focused ROCm/GGUF and AOT gates remain green.
What: add full-suite validation counts and classify residual failures in the consolidation plan.\n\nWhy: distinguish the fixed GGUF registry regression from missing dependencies, unsupported Triton/QSA paths, and CUDA-only Marlin tests.\n\nHow: record exact pass/skip/failure counts and preserve the 100-test change-scoped gate as the acceptance signal.\n\nValidation: plan diff passed git diff --check; focused GGUF/ROCm gate passed 100 tests and AOT coverage passed 2 tests.
Read split GGUF headers across every shard for Qwen expert metadata and resident sizing. Reject non-finite or out-of-range sampling penalties and benchmark timings, and keep gate metrics JSON-safe. Select HIP sonames numerically and align ROCm JIT linking with setup builds. Harden ROCm and NVIDIA workflows with selected-revision checkout, reusable venv bootstrap, safe target validation, and pinned mutable inputs. Verification: 170 focused tests passed; workflow YAML parsed; yamllint passed; compileall passed; git diff --check passed.,
What: - Remove .agents/learnings and .plans/rocm-consolidation files from the branch. - Remove internal increment and plan-path references from source comments and public installation docs. - Keep implementation comments that explain correctness, ownership, profiler intent, source attribution, or ROCm safety behavior. - Clarify public ROCm documentation: gfx1100 has recorded serving smoke on ROCm 7.2.1; the ROCm 7.14.x container is a reference environment, and other target cells remain compile-only until physical serving evidence exists. Why: - Keep merge surface focused on code, tests, reproducibility tooling, and user-facing documentation. - Prevent private planning history, review workflow language, stale plan paths, and local process notes from entering the upstream repository. - Avoid presenting compile success or a reference container as cross-target serving or performance proof. Related upstream work informing this branch: - PR FlashML-org#132: portable ROCm/HIP foundation. - PR FlashML-org#133: TVM-FFI index/store portability. - PR FlashML-org#135: RCCL tensor-parallel communication. - PR FlashML-org#136: native GGUF ROCm build and Q4_0 kernels. - PR FlashML-org#137: earlier AMD serving bring-up. - PR FlashML-org#217: source-fork ROCm, Qwen3.5 GGUF, and performance experiments. - PR FlashML-org#241: gfx1150 build, JIT, Triton, and attention hardening. - PR FlashML-org#260: gfx1151 validation and fallback/build evidence. - PR FlashML-org#316: HIP graph-capture-safe expert copies. - PR FlashML-org#378: CPU/Hybrid MoE graph replay safety. - Local branch milestones: 436263f, 926c1e8, e1d1856, 8a70c7e, and e5fd30f. Evidence: - 170 focused tests passed after cleanup. - gfx1100 is the only target with end-to-end Qwen3.5 GGUF serving smoke recorded here. - Remaining matrix targets are compile-only; no new throughput claim is published without a matching A/B manifest.
Separate ROCm GGUF JIT target resolution and cache diagnostics from CUDA paths while preserving CUDA dispatch behavior. Add explicit ROCm regression coverage, keep ROCm CI manual on self-hosted hardware, and run CUDA-safe host-extension validation inside an isolated CUDA container environment. Validation: - 54 no-GPU focused tests passed - 52 ROCm tests passed, 6 skipped on RX 7900 XTX gfx1100 - CUDA 13 host extensions compiled in isolated container - ROCm GGUF benchmark passed for Q4_K, Q5_K, Q6_K, and Q8_0 with matching output hashes - YAML, compileall, and diff checks passed Ref: FlashML-org#217
Consolidate PR-217 ROCm/GGUF runtime work into PR-132 while preserving generic ROCm fallback and CUDA isolation. - gate gfx1100 kernels by exact target and runtime evidence\n- fail closed when ROCm graph capture is unknown or unsupported\n- validate ROCm toolchain before kernel-cache builds\n- preserve HIP stream ordering for batch metadata copies\n- add replay identity and promotion-gate validation\n\nRefs: python/freetoken/kernel/gguf.py; python/freetoken/utils/arch.py; python/freetoken/engine/graph.py:138-142; python/freetoken/utils/graph_gate.py:331-410; python/freetoken/server/launch.py:171-181; freetoken-kernel-cache/build_backend.py:120-122; python/freetoken/kernel/csrc/jit/batch_memcpy.cuh:59-67.\n\nValidation: 37 passed, 8 skipped; compileall, py_compile, and diff checks passed.\nBenchmark: no valid current tok/s result; historical 34.69 tok/s is context only.\nFinal gate: NO-PROMOTION pending working Torch ROCm runtime and teacher-forced A/B benchmark.
Replace ABI-only self-tests with independent candidate-vs-reference numeric checks (status stays compile-only), tighten B10434 hidden stride to 512, add HIP/CUDA MoE shape/device/stride/alignment/expert-ID bounds, validate batch memcpy negative sizes and async cleanup, fix TP-safe local attention, replay v1 compat, prefill route metadata, the gfx1100 CLI alias, and ROCm kernel-cache target propagation. Assisted-by: pi
Assisted-by: pi
samuelishida
marked this pull request as draft
September 6, 2026 00:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.