Cache Arithmetic

A 32-layer transformer has 32 attention heads with head_dim=128, and uses fp16 (2 bytes/value). For one sequence of 4096 tokens at batch size 1, roughly how large is the KV cache?