6 / 9 Quantization & Low Precision

Key vs Value Granularity

Several KV-cache quantization schemes quantize cached keys with per-channel scales but cached values with per-token scales. This asymmetry exists mainly because:

A

Per-channel scales are cheaper to compute at decode time than per-token scales

B

Per-token scaling is mandatory for keys because attention softmax normalizes row-wise

C

Keys show large, consistent outlier magnitudes concentrated in specific channels across tokens, while value activations lack that structured outlier pattern

D

Values must always be stored at a higher bit-width than keys by convention

E

Keys are computed once per sequence while values get recomputed at every decode step