Inference 0/0

Inference

Training vs inference

Why serving a model differs from training it: generation one token at a time, the activations training keeps for the backward pass, FLOPs per token and the batch-size tradeoff. These differences decide where inference spends its memory and money.

0 / 7 solved

Questions

0/7 solved
Question Tags Difficulty Success
1. Same Weights, Same Prompt, Different Output
inferencedeterminism +3
Hard 50%
2. Activation Memory, With and Without Recompute
trainingactivation-memory +2
Hard —
3. When Serving Costs More Than Training
flopsscaling-laws +3
Hard 80%
4. Activation Amnesia
activation-memorybackpropagation +1
Medium —
5. One Token at a Time
autoregressive-generationinference-cost +1
Medium —
6. Cost Per Token, Split
flopstraining-vs-inference +1
Medium —
7. Batch Size Tug-of-War
batchingthroughput-latency-tradeoff +1
Medium 80%