Cost Per Token, Split

Per token processed, training a large language model is substantially more expensive in FLOPs than running inference on that same model. What is the main reason?

Training FLOPs scale with dataset size while inference FLOPs are fixed regardless of the model's parameter count

Training must recompute the tokenizer vocabulary embeddings on every step

Training requires a backward pass in addition to the forward pass, and the backward pass costs roughly twice the FLOPs of the forward pass alone

Inference tokens are always processed using a distilled, smaller copy of the trained model

Inference skips the attention mechanism entirely to save compute