Inference 0/0

Inference

KV cache and decoding

The keys and values a model keeps so each new token can attend to the past without recomputing it, how fast that cache grows, and how tokens are chosen from the output. The cache usually limits how many requests one GPU can serve at once.

0 / 10 solved

Questions

0/10 solved
Question Tags Difficulty Success
1. What's in the Vault
kv-cacheattention
Medium —
2. One Decode Step, Counted in Bytes
kv-cachedecode +2
Medium —
3. The Cache That Ate the GPU
kv-cachememory +1
Medium 40%
4. Cache Arithmetic
kv-cachememory
Medium 33%
5. Counting the Wasted Slots
paged-attentionkv-cache +3
Hard —
6. Roll the Dice or Don't
samplingdecoding +1
Hard —
7. Speculative Speedup, Actually
speculative-decodingacceptance-rate +1
Hard —
8. Draft and Verify
speculative-decodingrejection-sampling +1
Hard —
9. MLA's RoPE Workaround
mlakv-cache +1
Hard —
10. EAGLE Eats Medusa's Lunch
eaglemedusa +2
Insane —