Inference
Batching and serving
Prefill and decode, time to first token and inter-token latency, the roofline, continuous batching and disaggregated serving. Serving engines such as vLLM and SGLang are built around these tradeoffs.
Questions
Question Difficulty Success