llm_inference ignores temperature/top_p/top_k from the input JSON its own help tells you to set them in

[filipemartinsubrobotics]

Summary

examples/llm/llm_inference.cpp tells the user twice to put sampling parameters in the
input JSON, and then never reads them. temperature, top_p and top_k in the input
file are silently ignored, and generation is always greedy.

The runtime itself supports sampling — request.temperature, shouldUseNonGreedySampling()
and SamplingParams are all wired in cpp/runtime/llmRankRuntime.cpp — so only the example
driver is missing the parse.

Where the instruction is given

examples/llm/llm_inference.cpp:154

// For other sampling parameters (temperature, top_p, top_k), please specify them in the input JSON file

and again in --help, examples/llm/llm_inference.cpp:357

--maxGenerateLength       Override max generate length from input file
                          NOTE: For sampling parameters (temperature, top_p, top_k),
                          please specify them in the input JSON file instead of CLI

Those two comments are the only occurrences of temperature in the file that are not
talker_temperature (the TTS path, which is parsed, at line 935). Nothing assigns
request.temperature from inputData.

Reproduce

Same input file twice, changing only temperature, with a prompt long enough for sampling
to diverge:

{"batch_size": 1, "temperature": 0.3, "max_generate_length": 120,
 "requests": [{"messages": [{"role": "user", "content": [{"type": "text",
   "text": "Describe, in a few sentences, what a search and rescue robot should do when it enters an unfamiliar room."}]}]}]}

temperature 0.0, 3 runs -> 1 distinct output of 3
temperature 0.3, 3 runs -> 1 distinct output of 3

All six generations are byte-identical. With sampling active at 0.3, three runs would not be.

Why it matters beyond the missing feature

Sampling behaviour can differ qualitatively from greedy, so a benchmark taken through this
tool does not describe a deployment that samples. On this model (a 2.44 B Cosmos reasoner,
INT4-AWQ W4A16) at temperature 0.3 with top_p 0.9, open-ended generation enters a
non-terminating repetition loop that greedy decoding does not:

temperature 0.0   finish=stop     7 steps, coherent, terminates on its own
temperature 0.3   finish=length  83 steps, degenerates at step 12 and never recovers

and raising max_generate_length 250 -> 512 -> 2048 extends the loop (24 -> 83 -> 424
steps) without extending the coherent prefix past step 12. Anyone who measured that model
through llm_inference, having set "temperature": 0.3 in the input file as the help text
instructs, would have seen none of it and concluded the model was fine.

This is the same shape as the ENABLE_CUTE_DSL default that silently compiles the INT4
kernels out: a documented knob that does nothing, failing quietly rather than loudly.

Suggested fix

Parse the three fields where max_generate_length and batch_size are already read, and
populate the request. A warning when a recognised sampling key is present but unused would
also have surfaced this immediately.

Environment: DGX B300 (SM103), CUDA 13.0, TensorRT 10.13.2, Edge-LLM v0.10.1 (e8b2952).
Also present on the same file at main, by inspection.

---

Found during the NVIDIA / OpenHackathons / Oracle Open Models Codefest 2026, while
benchmarking an INT4-AWQ Cosmos3-Edge reasoner for an offline-first search-and-rescue
robotics entry (Team UBR Stack). The production target is a Jetson Orin Nano; the B300 is a
bench machine used for evaluation.

[filipemartinsubrobotics]

Correction: my diagnosis was wrong on two counts. The symptom is real and I have a root cause and a fix.

What I got wrong. I claimed llm_inference never reads the sampling fields from the input JSON. That is false. examples/utils/requestFileParser.cpp reads all three:

140:    float temperature = inputData.value("temperature", 1.0f);
141:    float topP = inputData.value("top_p", 0.8f);
142:    int64_t topK = inputData.value("top_k", 50);

and assigns them at 202-204, and llm_inference.cpp passes batchedRequests through unchanged. I grepped llm_inference.cpp, found the two comments, and stopped looking. The --help text is accurate. I also wrote "also present at main, by inspection" -- I never checked main; please disregard that line.

The actual cause. Both LLM sampler call sites omit the Philox offset:

cpp/runtime/decoding/vanillaDecoder.cpp:124   topKtopPSamplingFromLogits(..., context.stream);
cpp/runtime/llmRankRuntime.cpp:2581           topKtopPSamplingFromLogits(..., context.stream);

The declaration defaults it (sampling.h: uint64_t philoxSeed = 42, uint64_t philoxOffset = 0), so the offset is 0 at every decode step. In the kernel:

sampling.cu:636   curand_init(philoxSeed, batchIdx, philoxOffset, &localState);
sampling.cu:640   auto randNum = static_cast<float>(curand_uniform(&localState) * topP * sSum);

curand_init is keyed on the offset, so one uniform is drawn for the whole sequence. Sampling becomes deterministic and lands on the argmax wherever that token's mass exceeds the fixed draw -- which for a language model is almost every position. Temperature and top_p still scale the logits, so 0.0 and 0.3 select different deterministic paths, but neither samples.

Evidence. Four configurations, three runs each, 120 generated tokens, INT4-AWQ 2.44 B Cosmos reasoner:

temperature	top_p	distinct / 3	vs greedy
0.3	0.80	1	byte-identical
0.3	0.99	1	byte-identical
1.0	0.95	1	byte-identical
1.0	0.99	1	byte-identical

All twelve plus the greedy baseline share one sha256 over 679 characters. At temperature 1.0 with top_p 0.99 over 120 tokens, three runs diverging is near-certain; and a merely fixed seed would give a stable sampled trajectory that still differed from argmax. The sampler workspace is allocated, so the non-greedy path is entered and returns top-1 regardless.

What convinced me this is a bug rather than a choice: the TTS talker path already varies it deliberately -- qwen3OmniTTSRuntime.cpp:4085 passes /*philoxOffset=*/reprefillOffset under a comment about varying the offset per call. The requirement is understood in the codebase; the LLM decode loop just doesn't do it.

Fix, verified. A per-token offset on each call site, reset per sequence via the existing resetForNewSequences hook:

config	before	after
t=0.0	greedy	unchanged, byte-identical
t=0.3 p=0.80	identical to greedy	differs, reproducible across runs
t=1.0 p=0.95	identical to greedy	differs, reproducible across runs

Greedy is untouched, sampling samples, and a given input file still reproduces exactly.

Two things worth flagging on scope. Six TTS call sites have the same omission (qwen3OmniTTSRuntime.cpp 2186, 2446, 2558, 2660, 2760, 4326) -- I have not touched them, to keep this to one concern. And I added a unit test pinning the kernel's offset contract, but it passes on the unpatched tree: the kernel was never the faulty part, so it documents the invariant rather than guarding the regression. Catching a non-advancing caller needs a decoder-level test against a real runtime, which I have not written.

Environment: DGX B300 (SM103), CUDA 13.0, TensorRT 10.13.2, Edge-LLM v0.10.1 (e8b2952).