MLA's RoPE Workaround

DeepSeek-V2/V3's Multi-Head Latent Attention (MLA) caches a single shared low-rank latent vector per token instead of per-head K/V pairs, cutting KV cache size roughly 2-4x versus GQA (and ~9x versus MHA) at comparable or better quality. At inference, the up-projection matrix that expands this latent into per-head keys can be algebraically folded into the query projection, so full-size keys never need to be materialized or cached, only the tiny latent is. Why can't this same weight-absorption trick be applied to standard RoPE positional encoding, forcing DeepSeek to route a small separate 'decoupled' RoPE component per head outside the compressed path?

RoPE is only defined for query vectors, not key vectors, so keys always require an uncompressed path regardless of MLA

MLA discards positional information entirely to save memory, and the decoupled path reintroduces it via learned embeddings unrelated to RoPE

RoPE applies a position-dependent rotation between query and key, so the rotation sits between the two static projection matrices and prevents them from being pre-multiplied into one fixed matrix the way the content path can

RoPE needs full fp32 precision while the latent is stored in fp8, so the rotated components can't be packed into the compressed vector

The compressed latent isn't causally masked, so applying RoPE inside it would let tokens attend to future positions