Conversation
2 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds min-p sampling (HF
MinPLogitsWarper, vLLMmin_p) as a per-request knob, applied where HF applies it, after the repetition penalty and temperature and before top-k/top-p. Chatterbox's reference sampler runsmin_p=0.05there, and a model can't reproduce that outside the sampler because the penalty is applied inside it.SamplingReqConfig.min_p/SamplingConfig.min_p, default 0 (off).SamplerSpec.enable_min_p, default False, same idea asenable_repetion_penalty. Only nodes that opt in carry the filter in their captured sampler (a[B]buffer inSamplerBuffers, two extra passes over[B, V]per step). Every other node's graphs and eager path are untouched.apply_min_p(probs, min_p)dropsprobs < min_p * maxand renormalises. No CPU branches or data-dependent shapes, so it captures fine.min_p == 0rows and greedy one-hot rows pass through. It sits afterfused_temperature_softmaxand before FlashInfer's top-k/top-p. The XPU path masks the raw logits instead, since its kernel samples from logits.SamplerResource.ingest_requestrefusesmin_p > 0on a node that didn't opt in. Otherwise eager steps would honour it and graph replays wouldn't.How was it tested?
test/modular/test_sampler_min_p.pychecks the filter against the HF warper on random logits, identity at 0 and argmax at 1, greedy pass-through, the config reaching the eager sampler, buffers allocated only when enabled, the resource refusal and the spec default. One CUDA-only test (min_p=1is greedy for every seed on both the eager and graph paths) passed on an H100.docs/adding_models.rstdocuments the knob.ruff check .andpython -m pytest test/modularpass on the rebased branch.Checklist
ruff check .passes