Thirty Six Gigabytes

You are sizing a single GPU for a chat model with these dimensions:

property	value
transformer layers	48
query heads	64
key/value heads (grouped-query attention)	8
head dimension	128
KV dtype	bf16, 2 bytes per element

Every request holds a KV entry for both a key vector and a value vector, at every layer, for every token already in its context. You want to serve a batch of 32 concurrent sequences, each holding 6,144 tokens of context.

$\text{bytes} = 2 \times L \times h_{kv} \times d_{head} \times b \times s \times n$

How much GPU memory does the KV cache alone occupy?