Lesson 3 of 10
LLM serving: batching and the KV cache
Explain how batching and cached attention state affect serving capacity, memory use and response time.
In this lesson
- Weights plus KV cache
- Prefill and decode
- Batch ten requests
- Paging the KV cache
- Tensor and pipeline split
- Serving trade-offs
- What stays true
- Common mistakes
- Related topics and practice
Ultra Premium
Unlock this lesson
Ultra Premium opens the 10 premium lessons in AI systems, along with the rest of WhiteBox's premium lessons and problems.
Not ready yet? Read the free lesson: Backpressure