AI systems 0/10

Lesson 3 of 10

LLM serving: batching and the KV cache

Explain how batching and cached attention state affect serving capacity, memory use and response time.

1,589 words · 8 min read Key topic Ultra Premium Useful background: How an LLM call works, Back-of-envelope estimates

In this lesson

  1. Weights plus KV cache
  2. Prefill and decode
  3. Batch ten requests
  4. Paging the KV cache
  5. Tensor and pipeline split
  6. Serving trade-offs
  7. What stays true
  8. Common mistakes
  9. Related topics and practice

Ultra Premium

Unlock this lesson

Ultra Premium opens the 10 premium lessons in AI systems, along with the rest of WhiteBox's premium lessons and problems.

Not ready yet? Read the free lesson: Backpressure