Softmax on the Fly

Quantizing all attention weights to int8 before the matrix multiply to shrink memory bandwidth

Pruning low-magnitude attention scores to make the attention matrix sparse before the softmax is applied

Tiling Q, K, and V into blocks and computing attention with an online (running) softmax so the full N x N attention matrix is never materialized in HBM, cutting memory reads/writes

Replacing the softmax function with a cheaper ReLU-based approximation to reduce FLOPs

Precomputing and caching the entire attention matrix in HBM once per training run to avoid recomputation across steps