3 / 9 Inference Performance

Ridge Point Reading

On a roofline plot for LLM inference, a single decode step at batch size 1 lands with operational intensity far to the left of the ridge point (peak FLOPs / peak memory bandwidth). What does this tell you about the bottleneck for that step?

A

It is bound by inter-GPU network latency (NVLink/InfiniBand)

B

It is bound by CPU-side kernel launch overhead

C

It is memory-bandwidth bound: achievable throughput is capped by how fast weights can be streamed from HBM, not by compute throughput

D

It is compute bound: the GPU's tensor cores are saturated

E

It is bound by PCIe host-to-device transfer of the input tokens