FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness(opens in a new tab)
FlashAttention: compute exact attention in tiles that stay in on-chip SRAM, so the full attention matrix is never written to GPU memory, which cuts memory traffic and makes memory use linear in sequence length.