Skip to content

Capture the eager kernels for the qwen3tts talker decode - #310

Open
merceod wants to merge 1 commit into
mainfrom
fix-qwen3tts-instance-determinism
Open

merceod wants to merge 1 commit into
mainfrom
fix-qwen3tts-instance-determinism

Conversation

@merceod

@merceod merceod commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

The same seeded qwen3tts request comes back different on every server instance. Within one instance it is bitwise repeatable, across instances it is not, and greedy decoding differs too. The talker decode graph is captured from a torch.compile wrapper in max-autotune mode, which benchmarks kernel candidates in each process and keeps the fastest. Instances end up with different kernels, the rounding differs, and after a few dozen sampling steps the token sequence diverges. Nine instances on an H200 gave nine different outputs.

This captures the eager kernels instead (compile=False on the talker_decode config, which is what cosmos3 already does). Three fresh instances now give identical seeded and greedy output, and greedy matches the eager path.

The trade-off is speed. Decode is about 20 percent slower per request on an H200 (0.65 s instead of 0.53 s for a five second clip, 10.6 ms per codec frame instead of 8.8), and startup is about 30 s faster because inductor no longer runs for this walk. Batched throughput was not measured. Full pytest passes, and test_qwen3_tts_model now asserts the capture config.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant