6 / 11 Kernels (CUDA / Triton / Pallas)

One Trip, No Detours

A naive PyTorch implementation runs elementwise add, then a bias add, then a GELU activation as three separate CUDA kernel launches, each reading and writing the full tensor to HBM. What does fusing these into a single custom kernel primarily improve, and why?

A

It increases numerical precision by performing all arithmetic in float64 instead of float32

B

It converts the sequential computation into an embarrassingly parallel one across separate GPU streaming multiprocessors

C

It reduces memory-bound overhead by writing intermediate results to registers or shared memory instead of round-tripping through HBM between each op, and it cuts kernel launch overhead by combining three launches into one

D

It reduces the total number of floating point operations needed to compute the same mathematical result

E

It eliminates the need for the GPU to allocate any device memory at all