Training vs Inference

A trained model is a fixed function; inference is the forward-only generation loop. Where the compute goes.

0 / 7 solved
0/7 solved
Question Tags Difficulty Success
1. Same Weights, Same Prompt, Different Output
inferencedeterminism +3
Hard 67%
2. Activation Memory, With and Without Recompute
trainingactivation-memory +2
Hard —
3. When Serving Costs More Than Training
flopsscaling-laws +3
Hard —
4. Activation Amnesia
activation-memorybackpropagation +1
Medium —
5. One Token at a Time
autoregressive-generationinference-cost +1
Medium —
6. Cost Per Token, Split
flopstraining-vs-inference +1
Medium —
7. Batch Size Tug-of-War
batchingthroughput-latency-tradeoff +1
Medium —

Up next

7 of 7 questions here are still unsolved.

Next module NumPy & JAX