Skip to content

server: make exact prompt-cache hits generation-safe - #1887

Open
deyi2026 wants to merge 1 commit into
ml-explore:mainfrom
deyi2026:pr-1-exact-hit-generation-safe
Open

deyi2026 wants to merge 1 commit into
ml-explore:mainfrom
deyi2026:pr-1-exact-hit-generation-safe

Conversation

@deyi2026

Copy link
Copy Markdown

Problem

When a client resends the exact same full prompt, LRUPromptCache.fetch_nearest_cache
returns a legitimate exact hit with rest=[]. The empty segment then reaches
BatchGenerator.insert_segments and crashes the request (it needs at least one
prompt token to produce the first logits).

Fix

_generation_safe_cache_hit guarantees generation always has at least one
prompt token after a cache hit:

  • trimmable KV: trim one cached token and replay the final prompt token
  • hybrid recurrent caches (not trimmable as a whole): fall back to the nearest
    cache for prompt[:-1] and replay the final token
  • if no shorter cache exists: recompute the whole prompt (fail-safe)

Non-exact-hit paths are unchanged.

Testing

tests/test_server_cache_exact.py covers both cache kinds and the recompute
fallback (11 cases).

Keep at least one prompt token available for first-token logits on exact cache hits. Trim trimmable caches by one token; for non-trimmable hybrid caches reuse the nearest prelast checkpoint or safely recompute.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant