GPU and ML systems 0/0

GPU and ML systems

Parallelism and collectives

Splitting training across many accelerators with data, tensor, pipeline, expert and sequence parallelism, and the collectives that join them: all-reduce, reduce-scatter, all-gather and all-to-all. A frontier model does not fit on one chip, so every large training run combines several of these.

0 / 10 solved

Reading

  1. How to Scale Your Model(opens in a new tab)

    Austin et al. 2025 Online book

    An online book from Google DeepMind on how language models run on TPUs and GPUs: rooflines, how matrix multiplications are sharded across chips and what the communication costs, and how to split training and inference with data, tensor and other kinds of parallelism, with worked examples for Llama 3.