Skip to content

Add CPU-testable vLLM compaction boundary - #11

Merged
DaBestCode merged 2 commits into
mainfrom
codex/vllm-boundary
Sep 7, 2026
Merged

Add CPU-testable vLLM compaction boundary#11
DaBestCode merged 2 commits into
mainfrom
codex/vllm-boundary

Conversation

@DaBestCode

Copy link
Copy Markdown
Owner

Closes #7

Summary

  • pin the integration design to vLLM 0.28.0 / commit 2cf0a6915ce544dc493a0990f2ea38d81601128a
  • add a dependency-free, CPU-testable VLLMCompactionPlanner
  • preserve independent per-KV-head selection by emitting logical positions and source copy offsets
  • separate monotonic absolute token position from compacted physical length
  • document the proposed one-option call site, internal hook, upgrade risk, lifecycle behavior, and first-milestone exclusions
  • expose the planner and typed plan through the public package API

Design decision

vLLM's physical block table is shared across KV heads. Evicting whole blocks would force one retention mask across heads and silently change RandKV semantics. This boundary keeps selection head-specific and leaves block allocation plus PyTorch/CUDA copying to the runtime adapter.

Verification

  • ruff check .
  • ruff format --check .
  • mypy src
  • pytest -q — 25 passed
  • python -m build --no-isolation
  • git diff --check

Limitations

This does not patch or install vLLM and does not implement a CUDA kernel. It defines and tests the contract those pieces must consume.

@DaBestCode
DaBestCode merged commit 9a1f1b5 into main Sep 7, 2026
5 checks passed
@DaBestCode
DaBestCode deleted the codex/vllm-boundary branch September 7, 2026 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Design the minimal vLLM integration boundary

1 participant