Inference 0/0

Inference

Training vs inference

Why serving a model differs from training it: generation one token at a time, the activations training keeps for the backward pass, FLOPs per token and the batch-size tradeoff. These differences decide where inference spends its memory and money.

0 / 7 solved

Reading

Videos1

LLM Inference and GPU Performance Various speakers 1 video

Links1