GPU and ML systems 0/0

GPU and ML systems

Parallelism and collectives

Splitting training across many accelerators with data, tensor, pipeline, expert and sequence parallelism, and the collectives that join them: all-reduce, reduce-scatter, all-gather and all-to-all. A frontier model does not fit on one chip, so every large training run combines several of these.

0 / 10 solved

Questions

0/10 solved
Question Tags Difficulty Success
1. GPipe to 1F1B, What Actually Changes
pipeline-parallelism1f1b +3
Hard 40%
2. The Partial Sum Problem
parallelismtensor-parallelism +1
Medium 40%
3. Token, Meet Expert
moecollectives
Hard —
4. Ring's Constant Toll
collectivesring-all-reduce +1
Hard —
5. Bubble Trouble
pipeline-parallelismpipeline-bubble +1
Hard —
6. Softmax on a Rotating Ring
ring-attentionsequence-parallelism +2
Hard —
7. The NVLink Leash
bandwidthnvlink +1
Insane —
8. The 1.5x Tax
zerofsdp +1
Insane —
9. The 1.5x Tax
zero-3fsdp +2
Insane —
10. Four-Way Split, Zero Bubble
dualpipemoe +3
Insane —