Batch Size Compute Shift

A dense transformer's decode step reads roughly 2P bytes of fp16 weights per forward pass, independent of batch size B, while it performs roughly 2BP FLOPs (each of the B sequences reuses the same weights). As B increases, what happens to the step's operational intensity (FLOPs/byte), and what's the practical consequence?

Operational intensity falls with B because KV-cache reads scale faster than weight reads

Operational intensity falls because larger batches saturate memory bandwidth faster than compute can grow

Operational intensity rises roughly linearly with B, since weight bytes are amortized across more tokens, pushing the step from memory-bound toward compute-bound

Operational intensity stays flat because weights are re-fetched from HBM once per sequence in the batch

Operational intensity is unaffected by B since both FLOPs and bytes scale identically with batch size