Inference 0/0

Inference

KV cache and decoding

The keys and values a model keeps so each new token can attend to the past without recomputing it, how fast that cache grows, and how tokens are chosen from the output. The cache usually limits how many requests one GPU can serve at once.

0 / 10 solved

Reading

WhiteBox lessons1

Papers and essays1

  1. Efficient Memory Management for Large Language Model Serving with PagedAttention(opens in a new tab)

    Kwon et al. 2023 SOSP 2023 60 min

    PagedAttention and vLLM: store each request's KV cache in fixed-size blocks found through a block table, like pages of virtual memory, which nearly removes fragmentation and lets requests share blocks.

Videos1

LLM Inference and GPU Performance Various speakers 1 video

Links1