3 / 8 Mixture-of-Experts

Traffic Jam at Expert Seven

Experts are sharded one-per-device and tokens are routed to their assigned experts via an all-to-all dispatch, processed locally, then returned via an all-to-all combine. If one expert becomes 'hot' (receives far more tokens than the others that step), why does this slow down the entire training step rather than just that one device?

A

The hot expert's device must resize its parameter shard, which forces every other device to reshard its own expert weights

B

The all-to-all combine is a collective operation every device must wait on, so the step cannot finish until the device hosting the hot expert finishes its oversized batch, making the slowest expert the bottleneck for everyone

C

The hot expert's larger batch produces numerically unstable gradients that force an extra all-reduce pass across every device

D

Receiving more tokens causes the hot expert's device to recompute the router network, which re-triggers the router on all other devices too