4 / 10 Attention Internals

Fewer Keys, Same Heads

In grouped-query attention (GQA), a model with 32 query heads is configured with 8 KV groups. How many key/value head pairs must be stored in the KV cache per layer, and what happens to each of them during attention computation?

A

8 KV head pairs are stored, each one is duplicated internally back up to 32 before the dot product with queries

B

8 KV head pairs are stored; each is shared by 4 query heads (32/8), reducing KV cache size roughly 4x versus multi-head attention

C

32 KV head pairs are stored, same as multi-head attention, but only 8 are used during inference

D

1 KV head pair is stored and shared across all 32 query heads, identical to multi-query attention