1 / 7 Training vs Inference

Same Weights, Same Prompt, Different Output

An eval harness sends one identical prompt to a serving endpoint at temperature 0 all day and keeps getting different completions. The weights on disk never change, one GPU serves every request, and re-running a single matmul on that GPU with the same inputs is bitwise reproducible. The only thing that varies across the day is how many other users' requests share each forward pass.

What explains it, and what should be fixed?

A

Temperature 0 still samples from the softmax with a small numerical floor; setting top_k = 1 makes decoding genuinely greedy.

B

The server rotates between quantized and unquantized weight copies depending on load, so some requests run on a different numerical version of the model.

C

The attention, normalization, and matmul kernels are not batch invariant: their reduction order and split sizes change with batch size, so a token's logits differ in the last bits and eventually flip an argmax. The fix is batch-invariant kernels, not the sampler.

D

GPU floating-point arithmetic under concurrency is inherently nondeterministic, so identical inputs never give identical outputs and no kernel-level fix exists.

E

Blocks recycled between requests leave stale keys and values in the KV cache, so each run conditions on slightly different context.