8 / 9 GPU / TPU Architecture

Chasing the Ridge Point

Peak GPU tensor FLOPS have grown roughly 3x per generation (e.g. A100 to H100) while HBM bandwidth has grown only about 1.6-2x, pushing the roofline's compute/memory 'ridge point' further right each generation. To keep a fixed-FLOP-count GEMM or attention kernel compute-bound rather than memory-bound on newer hardware, which technique actually raises the kernel's arithmetic intensity (FLOPs executed per byte moved from HBM)?

A

Split the kernel into more, smaller thread blocks so the scheduler can better load-balance work across SMs

B

Increase the CTA/thread-block tile size so each operand tile staged into shared memory is reused across more multiply-accumulates before it is evicted

C

Increase the number of resident warps per SM so the scheduler has more warps to switch between while waiting on memory

D

Issue TMA bulk-copy loads earlier in the pipeline so global-memory latency overlaps with compute