Inference
KV cache and decoding
The keys and values a model keeps so each new token can attend to the past without recomputing it, how fast that cache grows, and how tokens are chosen from the output. The cache usually limits how many requests one GPU can serve at once.
Questions
Question Difficulty Success