2 / 9 Inference Performance

Where the Ridge Point Sits on an H100

NVIDIA's Hopper numbers for the H100 SXM5 are 1,000 TFLOPS of dense BF16 tensor-core throughput and 3 TB/s of HBM3 bandwidth. Arithmetic intensity is FLOPs executed per byte moved from HBM, and the roofline's ridge point is the intensity at which the bandwidth line meets the compute ceiling.

Where is the ridge point, and where does a batch-8 fp16 decode step at roughly 8 FLOPs per byte land?

A

About 333 FLOPs/byte, and the decode step at 8 FLOPs/byte is compute bound, since 8 FLOPs is far more work than moving 1 byte.

B

About 333 FLOPs/byte. At 8 FLOPs/byte the decode step is far to the left, memory-bandwidth bound, with a ceiling of 8 x 3 TB/s = 24 TFLOPS, about 2.4% of peak compute.

C

About 667 FLOPs/byte, using the 2,000 TFLOPS figure the same spec sheet quotes, with the decode step still to the left.

D

About 0.003 FLOPs/byte, so essentially every kernel including this one sits to the right and is compute bound.