6 / 10 KV Cache & Decoding

Roll the Dice or Don't

A production chatbot is looping on the same repetitive phrase, and switching it from greedy decoding to top-p sampling with a moderate temperature fixes it. What best explains why?

A

Greedy decoding samples randomly from the full vocabulary at every step, which is what causes the repetition

B

Greedy decoding always picks the single highest-probability token deterministically, which can lock the model into repetitive loops, while temperature reshapes the probability distribution and top-p samples from the smallest set of tokens whose cumulative probability exceeds p, introducing controlled randomness

C

Top-p sampling always selects exactly the k most probable tokens and picks among them uniformly at random

D

Temperature only affects the KV cache size, not the token selection process