GPU and ML systems
Parallelism and collectives
Splitting training across many accelerators with data, tensor, pipeline, expert and sequence parallelism, and the collectives that join them: all-reduce, reduce-scatter, all-gather and all-to-all. A frontier model does not fit on one chip, so every large training run combines several of these.
Questions
Question Difficulty Success