Inference 0/0

Inference

Batching and serving

Prefill and decode, time to first token and inter-token latency, the roofline, continuous batching and disaggregated serving. Serving engines such as vLLM and SGLang are built around these tradeoffs.

0 / 9 solved

Questions

0/9 solved
Question Tags Difficulty Success
1. Reading TTFT and ITL Off One Request
inferencelatency +3
Medium —
2. Where the Ridge Point Sits on an H100
rooflineridge-point +3
Hard —
3. Ridge Point Reading
rooflinememory-bound +1
Medium —
4. Tokens Per Second Off the Bandwidth Roof
decodebandwidth +3
Hard —
5. Batch Size Compute Shift
batchingoperational-intensity +1
Hard —
6. Keep the GPU Fed
batchingthroughput
Hard —
7. Split Personality
disaggregated-servingprefill-decode +1
Hard —
8. The P99 Penalty
continuous-batchingtail-latency +1
Insane —
9. Goodput, Not Just Grunt
continuous-batchinggoodput +1
Insane —