Skip to content

Conform Qwen3VL to PreparedInputSplitting - #627

Open
NivDvir wants to merge 1 commit into
ml-explore:mainfrom
NivDvir:feat/qwen3vl-split-conformance-v2
Open

NivDvir wants to merge 1 commit into
ml-explore:mainfrom
NivDvir:feat/qwen3vl-split-conformance-v2

Conversation

@NivDvir

@NivDvir NivDvir commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Proposed changes

Follow-up to #515. ChatSession reuses the KV cache on an append-only media turn only for models that conform to PreparedInputSplitting, and Qwen2.5-VL is the only one today, so Qwen3-VL prefills the whole conversation again each time a turn adds an image.

Qwen3VL can use the shared QwenVL.splitPreparedInput as is. Its vision attention is block-diagonal per media item, and deepstack features come from the pixels on every forward.

12 lines in Qwen3VL.swift, plus tests that check the split positions against Qwen3VL's own getRopeIndex and compare a cold full prefill with a split continuation for both conformers. Qwen3VLMoE isn't included because it has no offset-aware continuation path.

Checklist

Put an x in the boxes that apply.

  • I have read the CONTRIBUTING document
  • I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the necessary documentation (if needed)

AI usage

  • I have read this PR description in full and approve it as my own, and it
    accurately describes the code changes.
  • AI usage disclosure: Yes, I used Claude Code for parts of this. I reviewed and understand all of it.

Qwen3VL delegates to the shared QwenVL.splitPreparedInput with its own
image and video token ids and spatial merge size, so ChatSession can
reuse the KV cache on append-only media turns as it does for Qwen2.5-VL.

The shared split holds for this model because its vision attention is
block-diagonal per media item and deepstack features are rebuilt from
the pixels on every forward. Qwen3VLMoE does not conform because it has
no offset-aware continuation path.
@aleroot

aleroot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

This follow up sounds good to me. I checked the other VLM implementations, and I think there are three fairly low-hanging follow-ups that you could easily add to this pull request and reuse QwenVL.splitPreparedInput for image-only inputs:

  • Qwen35: it uses Qwen3VLVision, has the same concatenated patch-row + THW representation, and already has an offset-aware M-RoPE continuation path. Conforming Qwen35 should also cover Qwen35MoE, since it subclasses it.
  • GlmOcr: its processor has the same per-image row layout, vision attention is isolated using cuSeqlens, and its continuation path already carries the M-RoPE delta.
  • MuseGlimmer(very interesting model I am working on): its prepared image layout also matches the splitter’s actual contract, and both its full and window vision attention are split at per-image grid boundaries.

All three should still be gated by the same cold-full versus warm-split equivalence test. Gemma3 and LFM2VL also look feasible, but they need model-specific splitters because their pixels are arranged along a batch/image axis rather than as concatenated patch rows...

@NivDvir

NivDvir commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for going through the other models. I'd like to keep this PR to Qwen3VL so it stays a 12-line change that's quick to review.

I read Qwen35 and GlmOcr against the splitter and agree both fit. I can take them in a follow-up with the same cold-full versus warm-split test, or add them here if that's easier to review. MuseGlimmer is probably easier to add from your side, since you know its layout.

@aleroot

aleroot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

@NivDvir ok, I will port it to Muse Glimmer once your PRs are going to be merged. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants