Producers, Consumers, and Rotated Outliers

Every warp alternates between producer and consumer roles each iteration to hide TMA latency, and FP8 gets its accuracy from per-tensor (not per-block) scaling factors

Producer/consumer here refers to splitting attention heads across two GPUs, and the FP8 kernel runs the entire attention computation in FP8 with no rescaling step

Producer warp-groups issue async TMA loads of the next Q/K/V tile while consumer warp-groups run WGMMA matmuls and softmax on the current tile concurrently; FP8 quantizes per-block and multiplies Q/K by a random orthogonal (Hadamard-like) rotation first to spread outlier magnitudes before rounding

Producer warps precompute the softmax exponentials for the next tile while consumer warps handle only $QK^\top$, and FP8 accuracy is preserved purely by upcasting the final softmax normalization to FP32 with no quantization change

Producer warp-groups compute the $O = PV$ matmul while consumer warp-groups compute $QK^\top$ on a separate clock domain, and FP8 uses stochastic rounding in place of any quantization scaling