Inference
Training vs inference
Why serving a model differs from training it: generation one token at a time, the activations training keeps for the backward pass, FLOPs per token and the batch-size tradeoff. These differences decide where inference spends its memory and money.
Reading
Videos1
LLM Inference and GPU Performance Various speakers 1 video