LLM Serving & Inference Infrastructure

Prefill vs decode, the KV cache, continuous batching, streaming, GPU capacity and cold starts, and model routing.