Latent KV, Decoupled RoPE

DeepSeek's Multi-head Latent Attention (MLA) shrinks KV-cache memory versus standard multi-head attention. What is its core compression mechanism, and what complication does it force with rotary position embeddings?

MLA quantizes the KV cache to 4-bit per channel using a learned codebook; RoPE is applied after dequantization, so no architectural change to RoPE is required.

MLA shares one K/V projection across every transformer layer (cross-layer KV sharing); RoPE is computed once at the first layer and passed unchanged into every later layer's cache.

MLA down-projects keys and values into one shared low-rank latent vector per token (caching only that latent instead of per-head K/V), then up-projects per-head K/V from it on demand; since RoPE's rotation doesn't commute with this low-rank compression, MLA adds a small decoupled extra query/key path that carries RoPE directly instead of rotating the compressed latent.

MLA prunes attention heads at inference time based on a learned importance score and caches K/V only for the top-scoring heads; RoPE frequencies for pruned heads are redistributed across the remaining heads to preserve positional coverage.

MLA replaces the KV cache with a fixed-size recurrent state updated token by token, similar to a linear-attention state, which removes the need for RoPE entirely since position is encoded implicitly in the recurrence order.