One Trip, No Detours

A naive PyTorch implementation runs elementwise add, then a bias add, then a GELU activation as three separate CUDA kernel launches, each reading and writing the full tensor to HBM. What does fusing these into a single custom kernel primarily improve, and why?

It increases numerical precision by performing all arithmetic in float64 instead of float32

It converts the sequential computation into an embarrassingly parallel one across separate GPU streaming multiprocessors

It reduces memory-bound overhead by writing intermediate results to registers or shared memory instead of round-tripping through HBM between each op, and it cuts kernel launch overhead by combining three launches into one

It reduces the total number of floating point operations needed to compute the same mathematical result

It eliminates the need for the GPU to allocate any device memory at all