[Gemma4] vision-path A/B knobs for the 12B loop defect (bidi-on-full-layers, fp32 embedder) + NaN-safe softcap

[tsavo-at-pieces]

Gemma-4-12B generates degenerate repetition loops on vLLM GPU when given images.
This PR carries the two code changes that came out of a 13-arm investigation, plus
the full evidence trail. Both knobs default off; stock behavior is unchanged.

What's in the diff

1. VLLM_GEMMA4_BIDI_FULL_LAYERS=1 — keep bidirectional image spans on
   full-attention layers instead of clearing them. llama.cpp applies non-causal
   image attention on both its SWA and full KV caches; HF (and vLLM) document
   "causal only" for Gemma-4 global layers. Negative result — kept because
   reconstructing the A/B is a full fork-build-deploy cycle.
2. VLLM_GEMMA4_VISION_LN_FP32=1 / VLLM_GEMMA4_VISION_FP32=1 — fp32
   LayerNorm statistics, or the whole vision embedder in fp32. Ported narrowly
   from a sibling TPU stack's shipped fix. Negative result on CUDA (see below).
3. NaN-safe apply_softcap — a genuine hardening fix, unrelated to the loop
   defect. The helper computes x*tanh(S/x) via the exp form; fp32 exp
   overflows to inf past |S/x| ≈ 88.7, so the expression becomes inf/inf = NaN
   and poisons the whole softmax row. Now clamps S/x to ±30 first —
   tanh(±30) is exactly ±1.0 in fp32, so it is lossless. Shared by
   triton_unified_attention and int4_per_token_head. This one is worth
   taking on its own merits.

The finding that actually matters: max_soft_tokens=560

The defect scales with image-block size and disappears entirely below ~1024
soft tokens.

Config	image block	loops	parsed
max_soft_tokens=1120	~1094 tok	4/5 (3–4/5 across six independent arms)	1/5
max_soft_tokens=560	~534 tok	0 in 20 case-runs	9/10

At 560, per-case checks reach 20/20 and 22/22 — better reliability than the
llama.cpp reference (3/5 valid) on the same frames. The cost is fine-text
resolution: long URL ids and small headings degrade (one case 15/22).

Recommendation: anyone hitting gemma-4-12B degeneration on vision should set
max_soft_tokens=560 first. Any real fix must be validated at 1120
specifically — 560 hides the defect completely.

What was ruled out (13 arms)

Text-side, each by its own arm: attention backend, KV dtype (fp8 vs auto),
sampler (incl. min_p parity with llama-server), tokenizer/BOS/stop-set,
final_logit_softcapping, proportional RoPE. Decisive control: the same
server, weights, sampler and seed, given each case's content as OCR text
instead of a screenshot, produces zero loops in 5/5. The defect is
image-conditional.

Image-side, each patched, built, deployed and measured:

Hypothesis	Arm	Result
Bidi scope on full-attention layers	G	4/5 loops — no change
Preprocessing geometry / resampling (llama.cpp-parity client-side resize)	H	4/5 — no change
fp32 LayerNorm statistics	I	byte-identical to unpatched
Whole embedder in fp32	J	3/10 pooled vs stock 2/10 — noise
Sliding window clipping the block (widen 1024→2048)	L	10/10 — strictly worse
Bidi mask construction (disable mm_prefix entirely)	M	7/10 — no change

Two of these deserve comment:

• Arm L killed the most attractive hypothesis. sliding_window is 1024,
  sitting exactly between the two block sizes, so "the block is wider than the
  window and within-block bidirectional attention gets clipped" is the obvious
  story. Widening the window so the block fits made it worse — Gemma-4's
  sliding layers are trained at 1024 and degrade off-distribution. A source
  check agrees the composition isn't where engines differ: llama.cpp's
  LLAMA_SWA_TYPE_STANDARD masks iff p1 - p0 >= n_swa (one-sided, future KV
  never masked); this repo's Gemma-4 clamp is (query_abs_pos - seq_offset) < SLIDING_WINDOW. Same rule.
• Arm M exonerated the mask entirely. Every mask state loops at 1094 tokens
  (absent 7/10, stock 4/5, extended to all layers 4/5, window-widened 10/10) and
  none loops at 534. Note when reproducing: disabling mm_prefix makes
  FlashInfer eligible, so the backend silently changes unless you pin
  --attention-backend=TRITON_ATTN (and --disable-chunked-mm-input, which is
  otherwise auto-forced only when mm-prefix is on).

Where it stands: mask and numerics dimensions are exhausted; block size is
the only variable that moves the outcome. The untested remainder is
position-embedding / patch-ordering behavior at scale — the factorized 2D
posemb table and the patchify/space-to-depth ordering are the only vision-path
components whose behavior is a function of block geometry and that were never
varied independently of it.

Two reproduction constraints

• The HF processor accepts only max_soft_tokens ∈ (70, 140, 280, 560, 1120).
  A 960 arm dies at startup with an explicit ValueError, so there is no rung
  between 534 and 1094 tokens and the transition cannot be bracketed tighter.
• It upscales under-budget images to fill the budget: two case sets 6% apart
  in linear resolution returned byte-identical prompt-token counts. Block width
  is a function of max_soft_tokens alone, never of input resolution.

Environment / methodology note

vLLM nightly 46638857 (and f94666b6), RTX PRO 6000 (sm_120), BF16 and QAT
W4A16 checkpoints, 5 dense-screenshot cases, v6-mini free-form JSON,
temp 1.0 / top_p 0.95 / top_k 64, ≥2 seeds per arm.

Two measurement traps worth passing on, both of which nearly manufactured false
results:

• A failed Cloud Run deploy leaves the PREVIOUS revision serving, and the
  eval silently measures the old config. A 960-budget arm failed its startup
  probe and its "clean" numbers were really a third 560 run — caught only
  because the prompt-token counts were byte-identical. Gate every config arm on
  a value read back from the live response, never on "the deploy returned".
• VLLM_ATTENTION_BACKEND is removed upstream and only warns
  (Unknown vLLM environment variable detected). Deployments still setting it
  silently run auto-selection — ours did for four revisions, invalidating the
  backend label on an entire earlier result series. Assert the
  Using AttentionBackendEnum.X startup line instead.

Fork-first: nothing here has been filed upstream.

[cursor]

PR Summary

Medium Risk
The new fused SITU+FP8 CUDA path and gRPC LoRA lifecycle touch hot inference/MoE code paths; most other changes are CI, docs, and dead-code cleanup with lower runtime impact.

Overview
Adds agent skills for CUDA IMA debugging and GPU kernel microbenchmarking (workflow docs plus CUPTI and multi-GPU GEMM+reduce-scatter templates), with Claude symlinks into .agents.

CUDA / MoE: Extends libtorch-stable SITU activations with optional valid_rows (DeepEP token counts), refactors the math via situ_activation, and introduces situ_and_mul_quant—a fused activation plus block-FP8 (group size 128) path with a Kimi-K3–tuned pipelined kernel, hardware FP8 cvt, and padding-row scale fill. Python bindings and ops headers are updated accordingly. Flash-attention submodule GIT_TAG is bumped; unused GPTQ/CUTLASS/FP8 conversion helpers are trimmed.

Rust frontend: gRPC Control gains LoadLora / UnloadLora / ListLoras; Generate accepts lora_name with validation against engine LoRA support. LoraRequest construction is validated; LoRA errors use structured thiserror types.

DeepSeek V4 chat rendering: System (and trailing system) turns now emit the same assistant transition tokens as user turns in chat and thinking modes.

CI / Buildkite: Best-effort CRCR nightly job reporting (shell + Python JSON payloads, soft-fail). ROCm large-model lm-eval switches to models-large-rocm-tp4.txt, adds DeepSeek-V4-Flash-MXFP4 GSM8K config and tokenizer_mode wiring; MTEB moves MI250→MI300; entrypoints and model-runner-v2 distributed timeouts/tests adjusted.

Other: Docker rust-build split into cache (pretend version) + relink stage; batch_invariant CODEOWNERS path → determinism/; removes FireRedLID / some multimodal examples; docs refresh (XPU tables, supported-models reshuffle, Blackwell FA4 head_size=256 note, adaptive verification limitations).

^{Reviewed by Cursor Bugbot for commit 226145d. Configure here.}

[tsavo-at-pieces]

Update: vision-embedder precision arms added (both negative)

Second commit adds two env-gated knobs on Gemma4UnifiedForConditionalGeneration
and the measurements that close the vision-numerics theory on CUDA.

Background. A JAX/TPU deployment of this same gemma4_unified embedder was
independently found to corrupt near-flat image patches: torchax lowers
aten.native_layer_norm with bf16 mean/var, and for patches whose std is the
order of the bf16 quantization step, LayerNorm amplifies the bad rstd by
~1/std. Their fix computes the three embedder LayerNorms' statistics in fp32.
Ported here narrowly — LayerNorm submodules only, leaving upstream's embedder
forward and its ColumnParallelLinear call untouched (re-deriving that
plumbing is what broke their first attempt with a 6912-vs-3840 einsum
mismatch).

Result: it does not transfer, and neither does the stronger version.

Arm	Knob	parsed (5 dense screenshot cases)
I	VLLM_GEMMA4_VISION_LN_FP32=1	1/5 — byte-identical to unpatched
J	VLLM_GEMMA4_VISION_FP32=1 (whole embedder fp32)	2/5 @ seed 42, 1/5 @ seed 43 → 3/10 pooled vs stock 2/10

Arm I is a no-op by construction: PyTorch eager on CUDA already accumulates
LayerNorm statistics in fp32 for bf16 inputs. The TPU diagnostic's own stage
table says as much — at the soft-token output, torchax bf16 is rel 4.4e-2 /
cos 0.9944 while torch-eager bf16 (this path) is rel 3.1e-3 / cos 1.0000.
Their fix moves torchax onto the number CUDA already has.

Arm J closes even that residual 3.1e-3 and still does not help. Its seed-42 run
was briefly exciting — google-chat-65, which had looped identically across all
six prior vision arms, parsed at 16/22 — and the seed-43 resample put it back in
the loop. Against this model's ~40% intrinsic loop base rate, n=5 cannot resolve
a one-case flip; the pooled number is flat.

Engine logs confirm both knobs applied ([gemma4-unified] vision embedder LayerNorms ... fp32 / ... running in FP32 end-to-end), so these are true
negatives, not silent no-applies.

Net for triage: bf16 arithmetic in the vision embedder is not what drives
gemma-4-12B degeneration on CUDA. Combined with the earlier arms, the defect is
image-conditional with mask-level, preprocessing, and numeric explanations all
eliminated — and the two vLLM-family stacks turn out to have independent root
causes that merely present alike. Structural hypotheses remain open: soft-token
count/layout against the 1024 sliding window (~1160 soft tokens at a 1120
budget), and patchify / space-to-depth ordering.

Knobs default off; stock behavior unchanged. They are kept because this A/B is
otherwise a full fork-build-deploy cycle to reconstruct.

[tsavo-at-pieces]

Update 2: the defect scales with image-block size — max_soft_tokens=560 is clean

The largest effect found in this investigation, and it needs no patch at all.

Arm	Config	loops (cap-hits)	parsed
baseline	max_soft_tokens=1120 (~1094-token image block)	4/5, and 3-4/5 across six independent earlier arms	1/5
K	max_soft_tokens=560 (~534-token block)	0/20 case-runs	9/10
L	1120 + sliding_window 1024→2048	10/10 — worse	0/10

Arm K checks (seed 42 / 43): chrome-firebase 20/20 · 20/20, cursor-repo 19/20 ·
19/20, gdoc-wpe 15/22 · (stray \escape, not a loop), google-chat 22/22 ·
20/22, x-social 18/19 · 17/19. The one non-parse in ten is a bad escape
character in otherwise complete, fence-terminated JSON.

The obvious mechanism is wrong. text_config.sliding_window is 1024, which
sits neatly between the two block sizes, so "the block is wider than the window
and the within-block bidirectional attention gets clipped" is the natural
story. Arm L falsifies it: holding the block at 1094 and widening the window to
2048 — so the block fits — made the model strictly worse (10/10 loops).
Gemma-4's sliding layers are trained at 1024 and degrade off-distribution.

A source check says the same thing independently: llama.cpp's
LLAMA_SWA_TYPE_STANDARD masks iff p1 - p0 >= n_swa — one-sided, future KV
never masked — and this repo's gemma-4 clamp is (query_abs_pos - seq_offset) < SLIDING_WINDOW. Same rule; the window/block composition is not where the
engines differ.

Two reproduction constraints worth knowing. The HF processor accepts only
max_soft_tokens in (70, 140, 280, 560, 1120) — a 960 arm dies at startup with
an explicit ValueError — so there is no rung between 534 and 1094 tokens and
the transition cannot be bracketed tighter by budget. And it upscales
under-budget images to fill the budget: two case sets 6% apart in linear
resolution returned byte-identical prompt-token counts. Block width is a
function of max_soft_tokens alone, never of input resolution.

Where that leaves the bug. Image-conditional (text-only prompts are clean),
scales with block size, not the window, not the mask scope, not preprocessing,
not embedder precision. The remaining candidate is an error in the
bidirectional mm_prefix range construction whose cost grows with block size.
The decisive probe — which I have not run — is to disable mm_prefix
bidirectional attention entirely at budget 1120: if pure-causal is clean there,
the bug is in that mask path.

Practical upshot: anyone hitting gemma-4-12B degeneration on vision should
try max_soft_tokens=560 first. And any fix for the underlying bug must be
validated at 1120 specifically — 560 will hide it.

[Copilot]

Pull request overview

This PR primarily hardens GPU attention softcapping against NaNs and extends speculative/adaptive verification logprobs plumbing (including device-only request-boundary handling). However, the diff also includes substantial additional refactors and feature work (e.g., new LoRA lifecycle gRPC API, CLI/entrypoint reshuffles, model/quantization cleanups, and many test/documentation updates), making the effective scope significantly broader than the PR description.

Changes:

• Make Triton apply_softcap NaN-safe by clamping the exponent argument to a safe fp32 range.
• Enable logprobs with adaptive verification by carrying per-request boundaries as a tensor (device → async D2H) and threading that through logprobs tensors/lists.
• Add/extend parallelism features and supporting hooks (e.g., compute_logits_local / skip_gather path for batch-sharded sampling) plus a broad set of refactors, API, docs, and test updates across Python + Rust.

Reviewed changes

Copilot reviewed 224 out of 301 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
vllm/v1/worker/gpu/spec_decode/rejection_sampler.py	Plumbs adaptive-verification-aware request boundaries into logprobs computation.
vllm/v1/worker/gpu/spec_decode/adaptive_verification.py	Narrows backend/CG-support checks to selected layers and adds targeted CG-support querying.
vllm/v1/worker/gpu/sample/sampler.py	Factors logprobs dimension computation into a helper used by sharded gather paths.
vllm/v1/worker/gpu/sample/logprob.py	Allows cu_num_logits to be tensor-backed and stores device-only boundaries in outputs.
vllm/v1/worker/gpu/model_states/mamba_hybrid.py	Propagates aligned Mamba state indices to metadata builders when available.
vllm/v1/worker/gpu/model_states/encoder_decoder.py	Updates model-state docstring to reflect supported models.
vllm/v1/worker/gpu/attn_utils.py	Refactors cudagraph support detection into get_attn_cg_support and adds layer filtering.
vllm/v1/worker/gpu_worker.py	Updates import path for batch-invariance initialization.
vllm/v1/worker/gpu_model_runner.py	Adjusts logprobs tuple unpacking to tolerate extra returned fields.
vllm/v1/utils.py	Redacts sensitive args when logging Rust frontend launch command.
vllm/v1/outputs.py	Adds tensor-backed cu_num_generated_tokens and CPU materialization logic.
vllm/v1/engine/logprobs.py	Adjusts prompt logprobs tuple unpacking to tolerate extra returned fields.
vllm/v1/engine/core.py	Uses get_args(PauseMode) for pause-mode validation.
vllm/v1/core/sched/scheduler.py	Threads XD-RoPE flag into request scheduling and logs pause-state transitions.
vllm/v1/core/sched/output.py	Carries XD-RoPE flag through scheduler output request data.
vllm/v1/attention/ops/triton_attention_helpers.py	Clamps S/x before exp-form softcap to prevent fp32 inf/inf NaNs.
vllm/v1/attention/backends/mla/xpu_mla_sparse.py	Adds required decode/prefill counters to XPU sparse MLA metadata.
vllm/v1/attention/backends/mla/flashinfer_mla_sparse.py	Adds TRT-LLM metadata builder and declares cudagraph support for SM100 sparse MLA.
vllm/v1/attention/backends/flash_attn_diffkv.py	Initializes fa4_hd256 flag.
vllm/utils/b12x.py	Makes optional b12x module access more compile-safe via cached submodule imports.
vllm/utils/argparse_utils.py	Adds --task hinting for serve and overrides argparse error messaging.
vllm/triton_utils/importing.py	Excludes Triton “cpu” backend when counting active drivers to avoid false multi-driver detection.
vllm/transformers_utils/processors/cheers.py	Removes Cheers processor implementation.
vllm/transformers_utils/processors/init.py	Removes Cheers/FireRedLID processors from registry exports.
vllm/transformers_utils/processor.py	Simplifies compatibility docstring text.
vllm/transformers_utils/model_arch_config_convertor.py	Removes “umm” and MPT convertor wiring.
vllm/transformers_utils/configs/hyperclovax.py	Removes vendored HCXVisionConfig shim and related import.
vllm/transformers_utils/configs/fireredlid.py	Removes FireRedLID config class.
vllm/transformers_utils/configs/deepseek_vl2.py	Fixes DeepSeek-VL2 text config default vocab_size and aligns kv_lora_rank.
vllm/transformers_utils/configs/init.py	Removes multiple config registrations (e.g., Arctic/Cheers/FlexOlmo/HunYuan/HCXVision/FireRedLID).
vllm/transformers_utils/config.py	Removes config registry entries for removed model types.
vllm/transformers_utils/chat_templates/registry.py	Removes chameleon chat template fallback mapping.
vllm/tokenizers/deepseek_v4_encoding.py	Replaces asserts with ValueErrors and updates system-message generation transitions.
vllm/sampling_params.py	Lifts prior restriction disallowing output logprobs with adaptive verification.
vllm/profiler/wrapper.py	Fixes auto-stop to call public stop() so state fully resets; adds restart coverage.
vllm/parser/parser_manager.py	Simplifies parser composition logic (removes shared-engine shortcut behavior).
vllm/parser/gemma4.py	Expands tool-call parsing transitions to support parenthesis and tool-end transitions.
vllm/multimodal/utils.py	Preserves MM CPU metadata for XD-RoPE in prefix-cache stripping path.
vllm/multimodal/media/audio.py	Enforces strict base64 decoding and pre-checks local-file audio size before decode.
vllm/models/minimax_m3/nvidia/model.py	Adds compute_logits_local passthrough for local-logits path.
vllm/models/kimi_k3/nvidia/kda_metadata.py	Adds aligned Mamba indices field and uses it for “align” cache mode.
vllm/models/inkling/nvidia/logits_processor.py	Adds skip_gather plumbing; supports returning local logits without TP gather.
vllm/models/inkling/amd/logits_processor.py	Same as NVIDIA path: adds skip_gather support.
vllm/model_executor/models/transformers/utils.py	Adds maybe_per_layer helper for per-layer config fields.
vllm/model_executor/models/transformers/multimodal.py	Wraps image tokens with start/end markers for processors that require them.
vllm/model_executor/models/transformers/moe.py	Adds per-layer MoE config selection (top-k/intermediate/renorm) using extracted layer index.
vllm/model_executor/models/qwen3_omni_moe_thinker.py	Marks support for tower-connector LoRA.
vllm/model_executor/models/qwen3_dflash.py	Allows overriding decoder-layer class; uses class attr in layer construction.
vllm/model_executor/models/qwen3_5.py	Adds compute_logits_local (and wrapper passthrough) for local-logits path.
vllm/model_executor/models/llama.py	Updates doc comment and adds compute_logits_local.
vllm/model_executor/models/lfm2_vl.py	Marks support for tower-connector LoRA.
vllm/model_executor/models/jina_vl.py	Reworks Jina VL processing to keep cached MM ordering/hashes consistent with prompt template.
vllm/model_executor/models/granitemoehybrid.py	Tightens weight-loader expectations and types rotary-emb field.
vllm/model_executor/models/granite4_vision.py	Adds assertions/typing for multimodal embeddings in embed path.
vllm/model_executor/models/granite_speech.py	Strengthens audio dummy/options typing and audio input validation paths.
vllm/model_executor/models/gpt_neox.py	Adds assertion for intermediate tensors when provided path is taken.
vllm/model_executor/models/gpt_j.py	Adds assertion for intermediate tensors when provided path is taken.
vllm/model_executor/models/glmasr.py	Strengthens audio data handling and typing; avoids None/list ambiguity.
vllm/model_executor/models/glm4v.py	Makes MM grid extraction more defensive and fixes embed_input_ids signature forwarding.
vllm/model_executor/models/glm4_moe.py	Adds explicit moe_mlp_layers type annotation.
vllm/model_executor/models/glm4_moe_mtp.py	Asserts parallel_config is present before use.
vllm/model_executor/models/glm4_moe_lite.py	Cleans up q_lora_rank access, refactors mapping unpacking, and removes unused empty-tensor factory.
vllm/model_executor/models/glm4_moe_lite_mtp.py	Adds asserts for speculative config presence; adjusts overrides and mapping unpacking.
vllm/model_executor/models/glm_ocr.py	Accepts encoder_metadata and explicitly ignores it for compatibility.
vllm/model_executor/models/glm_ocr_mtp.py	Adds asserts for speculative config presence and refines KV-scale remap handling.
vllm/model_executor/models/gemma4_mtp.py	Adds speculative-config asserts and masked_embedding nullability assertions.
vllm/model_executor/models/gemma4_dspark.py	Adds assertion ensuring v_proj exists when k!=v path is used.
vllm/model_executor/models/gemma3n.py	Makes quant_config optional in signature.
vllm/model_executor/models/gemma3_mm.py	Adds path arg and asserts image input exists in eager encoder path.
vllm/model_executor/models/gemma.py	Adds assertion for intermediate tensors when used.
vllm/model_executor/models/deepseek_v2.py	Makes q_lora_rank optional and aligns projection dimension usage.
vllm/model_executor/models/conformer_encoder.py	Updates module docstrings to remove FireRedLID references.
vllm/model_executor/models/bailing_moe_v3.py	Handles missing mamba-family metadata during warmup/profile runs.
vllm/model_executor/models/apertus.py	Updates doc comment on bidirectional attention usage.
vllm/model_executor/layers/vocab_parallel_embedding.py	Moves batch-invariant import to determinism module path.
vllm/model_executor/layers/quantization/utils/w8a8_utils.py	Removes unused CUTLASS FP8 constant.
vllm/model_executor/layers/quantization/quark/schemes/quark_ocp_mx.py	Removes unused attribute assignments; keeps derived dtype logic.
vllm/model_executor/layers/quantization/quark/quark_moe.py	Removes unused float4 dtype probe.
vllm/model_executor/layers/quantization/inc/schemes/inc_scheme.py	Removes unused KV-cache quantization method stub and import.
vllm/model_executor/layers/quantization/fp8.py	Removes unused cutlass-block-fp8 support caching.
vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py	Removes unused cutlass-block-fp8 support caching/import.
vllm/model_executor/layers/logits_processor.py	Adds skip_gather flag to allow returning local logits before TP gather.
vllm/model_executor/layers/linear.py	Moves batch-invariant import to determinism module path.
vllm/model_executor/layers/layernorm.py	Moves batch-invariant import to determinism module path.
vllm/model_executor/layers/fused_moe/routed_experts.py	Docstring formatting/indentation cleanup.
vllm/model_executor/layers/fused_moe/prepare_finalize/deepep_v2.py	Carries prefix-sum metadata for cudagraph decode to support padding-row skipping.
vllm/model_executor/layers/fused_moe/modular_kernel.py	Allows missing expert counts and introduces valid_rows plumbed activation support.
vllm/model_executor/layers/fused_moe/experts/fused_batched_moe.py	Asserts expert counts are present for fused batched MoE apply path.
vllm/model_executor/layers/fused_moe/deep_gemm_utils.py	Falls back to local counting if expert counts are absent (carrier-only metadata).
vllm/model_executor/layers/fused_moe/activation.py	Adds valid_rows option and introduces fused SITU+FP8-quant helper wrapper.
vllm/model_executor/determinism/batch_invariant.py	Updates imports to new determinism config module path.
vllm/model_executor/determinism/init.py	Adds determinism package init.
vllm/lora/layers/logits_processor.py	Adds skip_gather arg and explicitly rejects skipping gather for lm_head LoRA.
vllm/envs.py	Clarifies audio filesize env var scope and enforcement points.
vllm/entrypoints/serve/utils/api_utils.py	Moves CLI args import, adds hf_token redaction, and exports redact_sensitive_args.
vllm/entrypoints/serve/dev/sleep/api_router.py	Removes stale FIXME comments about frontend multiprocessing behavior.
vllm/entrypoints/scale_out/render/serving.py	Adds Anthropic Messages render path converting to OpenAI chat before tokenization.
vllm/entrypoints/scale_out/render/api_router.py	Adds /v1/messages/render endpoint for Anthropic Messages requests.
vllm/entrypoints/launchers/run_batch.py	Fixes local imports for launcher entrypoint components.
vllm/entrypoints/launchers/render/entry.py	Switches to launcher CLI args module and reorders imports.
vllm/entrypoints/launchers/dp_supervisor.py	Fixes imports to use relative launcher modules.
vllm/entrypoints/launchers/cli_args.py	Rebrands CLI args module as “online server” and fixes import paths/constants.
vllm/entrypoints/launchers/api_server/entry.py	Switches to launcher CLI args module and reorders imports.
vllm/entrypoints/cohere/protocol.py	Bounds stop_sequences length via env limit using Pydantic Annotated Field.
vllm/entrypoints/cli/serve.py	Switches serve CLI to launcher CLI args and dp_supervisor module.
vllm/entrypoints/cli/launch.py	Switches launch CLI to launcher CLI args and reorders imports.
vllm/entrypoints/anthropic/serving.py	Renames conversion method to to_chat_completion_request and updates call sites.
vllm/engine/arg_utils.py	Adds --enable-batch-sharded-sampling plumbing into EngineArgs and config creation.
vllm/distributed/kv_transfer/kv_connector/v1/nixl/push_scheduler.py	Moves Mamba truncation hook out of cache-lookup path.
vllm/distributed/kv_transfer/kv_connector/v1/nixl/pull_scheduler.py	Same: moves Mamba truncation hook out of cache-lookup path.
vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_scheduler.py	Truncates Mamba requests on request admission (on_new_request).
vllm/distributed/device_communicators/all_reduce_utils.py	Adds FI/MNNVL allreduce max size for (103, 8, 1).
vllm/config/parallel.py	Adds enable_batch_sharded_sampling config field and documentation.
vllm/config/model.py	Adds uses_xdrope boolean property.
vllm/compilation/passes/fusion/allreduce_rms_fusion.py	Sizes FlashInfer allreduce+RMS workspace for max hidden dim across target+draft.
tools/pre_commit/mypy.py	Adjusts mypy exclusions to include more model files.
tests/weight_loading/models.txt	Removes deleted dummy model entry; keeps FP8 model.
tests/v1/worker/test_gpu_rejection_sampler_chunking.py	Sets new enable_adaptive_verification attribute in test fixture.
tests/v1/worker/test_gpu_profiler.py	Adds regression tests for restart after max-iterations auto-stop.
tests/v1/worker/test_gpu_model_runner_v2_eplb.py	Adds enable_batch_sharded_sampling to runner parallel config stub.
tests/v1/test_outputs.py	Adds coverage for tensor-backed request boundaries in LogprobsTensors.tolists.
tests/v1/spec_decode/test_adaptive_verification.py	Adds tests for layer-scoped backend/CG checks without weakening runner CG mode.
tests/v1/kv_connector/unit/test_nixl_push_connector.py	Adds test ensuring P-side Mamba truncation happens before cache lookup.
tests/v1/kv_connector/unit/test_nixl_connector_hma.py	Updates truncation test to use on_new_request and checks idempotency.
tests/v1/engine/test_output_processor.py	Adjusts unpacking for changed prompt-logprobs tensor shape/fields.
tests/v1/determinism/test_rms_norm_batch_invariant.py	Updates import path for determinism batch-invariant RMSNorm.
tests/v1/determinism/test_matmul_batch_invariant.py	Updates import paths and expands test to transposed-B cases.
tests/v1/core/utils.py	Ensures EC producer role enables minimal multimodal config in scheduler fixture.
tests/v1/core/test_output.py	Adds XD-RoPE coverage for strip_covered_mm_data.
tests/utils_/test_tensor_schema.py	Replaces removed HCX schema with a minimal local double-nested schema.
tests/tools/test_docker_build_metadata_args.py	Verifies rust build cache excludes .git in cached stage and relinks for exact version.
tests/tokenizers_/test_detokenize.py	Removes reference to deleted MPT model.
tests/test_triton_utils.py	Adds tests covering cpu backend behavior in Triton driver detection.
tests/test_audio_media_size_precheck.py	Adds load_bytes/load_file oversize checks and fixes header case for Content-Length.
tests/renderers/test_hf.py	Removes chameleon from HF content-format resolve test list.
tests/plugins_tests/test_endpoint_plugins.py	Updates import path for launcher CLI args.
tests/parser/engine/test_parser_engine.py	Updates expectations around parser manager behavior and reasoning wiring.
tests/multimodal/media/test_audio.py	Adds test that malformed base64 is rejected with ValueError.
tests/models/test_registry.py	Adjusts terratorch-only gating.
tests/models/test_initialization.py	Adjusts terratorch-only gating.
tests/models/multimodal/processing/test_qwen2_vl.py	Adds Jina VL processing order/cache alignment regression test.
tests/models/multimodal/processing/test_common.py	Removes prior skip for jina-reranker-m0 processing correctness.
tests/models/multimodal/generation/test_common.py	Removes chameleon VLM test entries.
tests/models/language/pooling_mteb_test/test_modernbert_fp8.py	Refines ROCm FP8 skip logic based on GPU kernel availability.
tests/model_executor/kernels/test_b12x_linear.py	Adds dynamo/compile safety test for b12x submodule lookup.
tests/kernels/moe/test_moe.py	Updates activation test call to pass new valid_rows arg.
tests/evals/gsm8k/configs/humming/Qwen3-30B-A3B-MXFP4A16-humming-act-fp8-block.yaml	Adds new eval config for block-FP8 activation quant.
tests/evals/gsm8k/configs/humming/config-act-fp8.txt	Registers new humming eval config.
tests/entrypoints/serve/utils/test_api_utils.py	Adds hf_token redaction coverage and updates redaction API name.
tests/entrypoints/pooling/scoring/test_cross_encoder_online_vision.py	Updates expected FLEX_ATTENTION threshold.
tests/entrypoints/launchers/test_dp_supervisor.py	Updates import path to launcher dp_supervisor module.
tests/entrypoints/launchers/test_cli_args.py	Updates import paths and test utilities import location.
tests/entrypoints/launchers/api_server/test_api_server_process_manager.py	Adds test ensuring Rust frontend launch logs redact credentials.
tests/entrypoints/cohere/test_registry_and_args.py	Updates import path to launcher CLI args.
tests/entrypoints/cohere/test_protocol.py	Adds stop_sequences limit validation test.
tests/entrypoints/anthropic/test_anthropic_messages_conversion.py	Updates docs/tests to use to_chat_completion_request rename.
tests/engine/test_arg_utils.py	Updates dp_supervisor import paths.
tests/distributed/test_pipeline_parallel.py	Removes deleted Arctic/MPT and chameleon entries.
tests/distributed/test_comm_ops.py	Updates expected mnnvl workspace size for (103, 8, 1).
tests/config/test_multimodal_config.py	Adds test: EC producer-only enables mm_encoder_only.
tests/config/test_model_arch_config.py	Removes commented reference to deleted MPT model.
tests/config/base_model_arch_groundtruth.json	Removes groundtruth entry for deleted mosaicml/mpt-7b.
tests/compile/passes/distributed/test_fusion_all_reduce.py	Adds tests for _fused_ar_workspace_hidden_dim sizing across target/draft.
rust/src/server/src/state.rs	Adds LoRA-enabled checks and exposes list/load/unload operations for lifecycle API.
rust/src/server/src/routes/tests.rs	Adds ready-response customization helper for admin app tests.
rust/src/server/src/grpc/inference.rs	Threads lora_name through spans and resolves/validates LoRA for gRPC inference requests.
rust/src/server/Cargo.toml	Adds thiserror dependency.
rust/src/engine-core-client/src/protocol/lora.rs	Makes LoraRequest::new validated (Result-returning) with typed errors.
rust/src/chat/src/renderer/deepseek_v4/tests.rs	Adds system→assistant transition tests for thinking/chat modes.
rust/src/chat/src/renderer/deepseek_v4/encoding.rs	Allows assistant transition after system messages where appropriate.
rust/proto/inference.proto	Adds lora_name field to GenerateRequest.
rust/proto/control.proto	Adds LoRA lifecycle RPCs and messages to Control service.
rust/Cargo.lock	Records thiserror dependency addition.
requirements/test/nightly-torch.txt	Updates einops comment after MPT removal.
examples/rl/rlhf_async_new_apis.py	Updates batch-invariant import path.
examples/generate/multimodal/vision_language_multi_image_offline.py	Removes HyperCLOVAX seed-vision example and registry entry.
examples/generate/multimodal/encoder_decoder_multimodal_offline.py	Removes FireRedLID example wiring.
examples/generate/multimodal/audio_language_offline.py	Removes FireRedLID example wiring.
docs/usage/security.md	Clarifies audio filesize limit enforcement across input types.
docs/models/pooling_models/embed.md	Removes GritLM entry.
docs/mkdocs/gen_files/generate_argparse.py	Updates argparse generator to mock launcher cli_args module.
docs/features/speculative_decoding/adaptive_verification.md	Updates limitations: removes “output logprobs not supported” note.
docs/design/attention_backends.md	Documents Blackwell FA4 head_size=256 constraints and fallbacks.
docs/contributing/model/multimodal.md	Removes chameleon reference from examples.
docker/Dockerfile.xpu	Splits Rust build into cached stage + relink stage for exact git-derived versioning.
docker/Dockerfile.cpu	Same: Rust build cache stage + relink stage for exact versioning.
docker/Dockerfile	Same: Rust build cache stage + relink stage for exact versioning.
csrc/libtorch_stable/torch_bindings.cpp	Extends situ op signature and registers new situ_and_mul_quant op.
csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm.cuh	Removes unused/obsolete SM100 gemm template specialization.
csrc/libtorch_stable/quantization/machete/machete_prepacked_layout.cuh	Removes unused NKL layout helper.
csrc/libtorch_stable/quantization/gptq/qdq_util.cuh	Removes unused dq_scale helper.
csrc/libtorch_stable/quantization/gptq/matrix_view.cuh	Removes unused half2 helpers and q4 column view.
csrc/libtorch_stable/quantization/gptq_allspark/allspark_utils.cuh	Removes unused cp_async_ca helper.
csrc/libtorch_stable/ops.h	Updates situ signatures and declares situ_and_mul_quant.
cmake/external_projects/vllm_flash_attn.cmake	Bumps flash-attention submodule GIT_TAG.
.github/CODEOWNERS	Updates batch-invariant ownership path to determinism module directory.
.buildkite/test-amd.yaml	Moves MTEB lane to MI300; updates lm-eval config list filename.
.buildkite/test_areas/model_runner_v2.yaml	Increases timeout and adds sharded sampling e2e tests to MRv2 distributed lane.
.buildkite/test_areas/entrypoints.yaml	Increases entrypoints unit timeout.
.buildkite/test_areas/crcr_report.yaml	Adds CRCR nightly report step group.
.buildkite/lm-eval-harness/test_lm_eval_correctness.py	Adds tokenizer_mode passthrough into lm-eval harness args.
.buildkite/lm-eval-harness/configs/models-large-rocm-tp4.txt	Adds DeepSeek-V4-Flash MXFP4 eval config.
.buildkite/lm-eval-harness/configs/DeepSeek-V4-Flash-MXFP4.yaml	Adds ROCm TP4 eval config including deepseek tokenizer mode and required arch gating.
.agents/skills/kernel-microbenchmark/benchmarks/cupti_microbenchmark.py	Adds CUPTI timing microbenchmark template.
.agents/skills/kernel-microbenchmark/agents/openai.yaml	Registers kernel microbenchmark skill metadata.
.agents/skills/debug-ima/SKILL.md	Adds debug-ima skill documentation stub.
.agents/skills/debug-ima/agents/openai.yaml	Registers debug-ima skill metadata.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[tsavo-at-pieces]

Final confirmation probe (2026-08-27), closing the investigation: grammar-constrained decoding does not mitigate the 1120 loops. xgrammar + `disable_any_whitespace` at `max_soft_tokens=1120`: 15/20 loops vs 4/5 free-form control, same revision, same image digest (nightly-46638857) as the arm-K 560 validation.

Mechanism: the repetition lives inside JSON string literals (escaped-`\n` runs, `☺` spam, phrase repetition) — grammar-legal until the token cap truncates. The defect is in the vision-conditioned next-token distribution, below the decode-constraint layer; no grammar/sampler/mask/KV/backend knob reaches it.

Production recommendation unchanged and now final: `max_soft_tokens=560` on GPU. Full table in the bench repo (BENCH_RESULTS.md, 2026-08-27).

🤖 Generated with Claude Code