Key vs Value Granularity

Per-channel scales are cheaper to compute at decode time than per-token scales

Per-token scaling is mandatory for keys because attention softmax normalizes row-wise

Keys show large, consistent outlier magnitudes concentrated in specific channels across tokens, while value activations lack that structured outlier pattern

Values must always be stored at a higher bit-width than keys by convention

Keys are computed once per sequence while values get recomputed at every decode step