7 / 10 Attention Internals

KV Bytes Per Token: MHA, GQA, MLA

DeepSeek-V2's Table 1 gives per-token KV cache in elements: MHA 2nhdhl2 n_h d_h l, GQA 2ngdhl2 n_g d_h l, MLA (dc+dhR)l(d_c + d_h^R) l. The released model has l=60l = 60 layers, nh=128n_h = 128 heads, dh=128d_h = 128, dc=4dh=512d_c = 4 d_h = 512, dhR=dh/2=64d_h^R = d_h / 2 = 64, all cached in fp16.

What does one token cost under MLA on this shape, and how does that compare with 8-group GQA?

A

About 8.8 MB per token, since each of the 128 heads caches its own 576-element latent.

B

About 69 KB per token, but only with RoPE disabled, since rotary keys cannot be compressed into the latent at all.

C

About 69 KB per token, roughly 3.6x less than GQA-8's 246 KB and about 57x less than MHA's 3.9 MB. The paper describes the result as matching GQA with only 2.25 groups.

D

About 138 KB per token, since the latent must be cached once for keys and once for values.

E

About 246 KB per token, the same as GQA-8: MLA shifts the cost from memory to compute without shrinking the bytes stored.