Skip to content

Chatterbox/Chatterbox-Turbo PR 2: min_p as a per-request sampler knob - #304

Open
merceod wants to merge 5 commits into
mainfrom
engine/sampler-min-p
Open

merceod wants to merge 5 commits into
mainfrom
engine/sampler-min-p

Conversation

@merceod

@merceod merceod commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds min-p sampling (HF MinPLogitsWarper, vLLM min_p) as a per-request knob, applied where HF applies it, after the repetition penalty and temperature and before top-k/top-p. Chatterbox's reference sampler runs min_p=0.05 there, and a model can't reproduce that outside the sampler because the penalty is applied inside it.

  • SamplingReqConfig.min_p / SamplingConfig.min_p, default 0 (off).
  • SamplerSpec.enable_min_p, default False, same idea as enable_repetion_penalty. Only nodes that opt in carry the filter in their captured sampler (a [B] buffer in SamplerBuffers, two extra passes over [B, V] per step). Every other node's graphs and eager path are untouched.
  • apply_min_p(probs, min_p) drops probs < min_p * max and renormalises. No CPU branches or data-dependent shapes, so it captures fine. min_p == 0 rows and greedy one-hot rows pass through. It sits after fused_temperature_softmax and before FlashInfer's top-k/top-p. The XPU path masks the raw logits instead, since its kernel samples from logits.
  • SamplerResource.ingest_request refuses min_p > 0 on a node that didn't opt in. Otherwise eager steps would honour it and graph replays wouldn't.

How was it tested?

test/modular/test_sampler_min_p.py checks the filter against the HF warper on random logits, identity at 0 and argmax at 1, greedy pass-through, the config reaching the eager sampler, buffers allocated only when enabled, the resource refusal and the spec default. One CUDA-only test (min_p=1 is greedy for every seed on both the eager and graph paths) passed on an H100. docs/adding_models.rst documents the knob. ruff check . and python -m pytest test/modular pass on the rebased branch.

Checklist

  • ruff check . passes
  • Added or updated tests / docs where relevant

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant