Conversation
Qwen3VL delegates to the shared QwenVL.splitPreparedInput with its own image and video token ids and spatial merge size, so ChatSession can reuse the KV cache on append-only media turns as it does for Qwen2.5-VL. The shared split holds for this model because its vision attention is block-diagonal per media item and deepstack features are rebuilt from the pixels on every forward. Qwen3VLMoE does not conform because it has no offset-aware continuation path.
|
This follow up sounds good to me. I checked the other VLM implementations, and I think there are three fairly low-hanging follow-ups that you could easily add to this pull request and reuse QwenVL.splitPreparedInput for image-only inputs:
All three should still be gated by the same cold-full versus warm-split equivalence test. Gemma3 and LFM2VL also look feasible, but they need model-specific splitters because their pixels are arranged along a batch/image axis rather than as concatenated patch rows... |
|
Thanks for going through the other models. I'd like to keep this PR to Qwen3VL so it stays a 12-line change that's quick to review. I read Qwen35 and GlmOcr against the splitter and agree both fit. I can take them in a follow-up with the same cold-full versus warm-split test, or add them here if that's easier to review. MuseGlimmer is probably easier to add from your side, since you know its layout. |
|
@NivDvir ok, I will port it to Muse Glimmer once your PRs are going to be merged. Thanks. |
Proposed changes
Follow-up to #515.
ChatSessionreuses the KV cache on an append-only media turn only for models that conform toPreparedInputSplitting, and Qwen2.5-VL is the only one today, so Qwen3-VL prefills the whole conversation again each time a turn adds an image.Qwen3VLcan use the sharedQwenVL.splitPreparedInputas is. Its vision attention is block-diagonal per media item, and deepstack features come from the pixels on every forward.12 lines in
Qwen3VL.swift, plus tests that check the split positions against Qwen3VL's owngetRopeIndexand compare a cold full prefill with a split continuation for both conformers.Qwen3VLMoEisn't included because it has no offset-aware continuation path.Checklist
Put an
xin the boxes that apply.pre-commit run --all-filesto format my code / installed pre-commit prior to committing changesAI usage
accurately describes the code changes.