6 / 7 Training vs Inference

Cost Per Token, Split

Per token processed, training a large language model is substantially more expensive in FLOPs than running inference on that same model. What is the main reason?

A

Training FLOPs scale with dataset size while inference FLOPs are fixed regardless of the model's parameter count

B

Training must recompute the tokenizer vocabulary embeddings on every step

C

Training requires a backward pass in addition to the forward pass, and the backward pass costs roughly twice the FLOPs of the forward pass alone

D

Inference tokens are always processed using a distilled, smaller copy of the trained model

E

Inference skips the attention mechanism entirely to save compute