Question 732

One Decode Step, Counted in Bytes

The vLLM paper works out the KV cache of a 13B OPT model at 800 KB per token: 2 (key and value) x 5,120 (hidden size) x 40 (layers) x 2 bytes (fp16).

A request on that model already has 2,000 tokens cached and is about to generate token 2,001. Counting only KV-cache traffic for this single decode step, how much is read and how much is written?

A

Read about 1.6 GB and write about 1.6 GB, since every cached entry is rewritten with the updated attention output.

B

Read 0 bytes: the attention scores over past tokens were computed during prefill and cached, so only the new query is needed.

C

Read about 1.6 GB, every cached token's keys and values at every layer, and write 800 KB, the one new token's keys and values.

D

Read 800 KB and write 800 KB: the cache exists so that only the newest token's keys and values are touched.

E

Read about 40 MB, one layer's worth, because all layers attend over the same shared cache.