1 / 10 KV Cache & Decoding

What's in the Vault

During autoregressive generation with a transformer, what exactly does the KV cache store?

A

A compressed copy of the model's weight matrices for faster matmuls

B

The key and value projections computed for every previously processed token, at every layer and head

C

The query vectors for the current token being generated

D

The final softmax probabilities output for each generated token so far