Question 733

The Cache That Ate the GPU

A team doubles both the context length and the number of transformer layers in their model, keeping everything else fixed. Roughly how does the KV cache's memory footprint change?

A

It grows about 2x, since only sequence length affects cache size

B

It shrinks, since more layers let the model compress earlier tokens' representations

C

It grows about 4x, since cache size scales linearly with both sequence length and number of layers (and also with number of heads)

D

It stays the same, because the KV cache size only depends on the model's hidden dimension

E

It grows exponentially, since attention has quadratic memory cost in sequence length