6 / 8 Mixture-of-Experts

All-to-All Wall

When scaling a Mixture-of-Experts model across many GPUs with expert parallelism, throughput often plateaus even though per-GPU FLOPs stay low. What is the primary systems-level cause?

A

Gradient all-reduce for the shared attention layers becomes serialized behind expert computation

B

The all-to-all communication that scatters tokens to their assigned experts' devices and gathers results back becomes bandwidth-bound across the interconnect

C

The router's softmax computation becomes the dominant cost as expert count grows

D

Each expert must store a full copy of the embedding table, exhausting HBM