One Token at a Time

A language model generates the sentence "The cat sat" one token at a time. Why does this autoregressive loop mean inference still burns significant compute even though weights are frozen and no gradients are computed?

Because tokenization itself is more expensive computationally than the forward pass

Each new token requires a fresh forward pass conditioned on all previous tokens, so generating N tokens means running the forward computation roughly N times (even with caching, each step still does real matrix multiply work)

Because frozen weights force the model to recompute backpropagation for every generated token to verify correctness

Because the model must retrain itself briefly on each partial sentence before producing the next token