4 / 9 Inference Performance

Tokens Per Second Off the Bandwidth Roof

A 70B-parameter dense model is quantized to fp8 and served on a single H100 SXM5: 80 GB of HBM3 at 3 TB/s. At batch size 1 with a short context, each decode step has to stream essentially every weight from HBM to produce one token.

Ignoring KV-cache traffic, what is the ceiling on tokens/sec, and what does a measured 28 tokens/sec tell you?

A

About 14,000 tokens/s: the step is compute bound, so the ceiling is NVIDIA's 2,000 TFLOPS dense fp8 peak for this part divided by the 2 x 70e9 FLOPs a forward pass costs per token.

B

About 43 tokens/s, and batching cannot raise it, because the ceiling is defined per token.

C

About 43 tokens/s: 3 TB/s divided by 70 GB of fp8 weights. Measuring 28 means the kernels already reach roughly two thirds of the bandwidth roof, so the remaining headroom is in memory efficiency and batching, not in FLOPS.

D

About 21 tokens/s: 3 TB/s divided by 140 GB, two bytes per parameter.

E

About 43,000 tokens/s, reading the bandwidth as 3,000 GB/s against 70 MB of weights per layer.