GPU and ML systems 0/0

GPU and ML systems

Kernels

Writing GPU kernels in CUDA, Triton and Pallas: launch geometry, tiling, fusion, occupancy and the online softmax behind FlashAttention. A fused, tiled kernel avoids round trips to slow memory, which is where FlashAttention’s speedup comes from.

0 / 11 solved

Reading

Papers and essays1

  1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness(opens in a new tab)

    Dao et al. 2022 NeurIPS 2022 75 min

    FlashAttention: compute exact attention in tiles that stay in on-chip SRAM, so the full attention matrix is never written to GPU memory, which cuts memory traffic and makes memory use linear in sequence length.

Videos3

LLM Inference and GPU Performance Various speakers 2 videos

Jane Street Tech Talks: Machine Learning Jane Street 1 video

Links2