Activation Amnesia

A transformer's forward pass is identical at training time and at inference time, yet training consumes far more GPU memory for the same batch and sequence length. What is the primary cause?

Training requires a bigger KV cache than inference because gradients are cached per token

Training must retain intermediate layer activations for the backward pass; inference can discard each layer's activations once the next layer consumes them

Training uses a larger vocabulary embedding table than inference

Inference always runs in a lower numerical precision than training, which is what shrinks memory