8 / 10 Parallelism & Collectives

The 1.5x Tax

ZeRO stage 3 (fully sharding optimizer states, gradients, and parameters) is compared against a standard data-parallel baseline that only does a single all-reduce of gradients per step. Why does ZeRO-3 incur roughly 1.5x the communication volume of that baseline instead of matching it?

A

It duplicates the gradient reduce-scatter twice per step to guarantee fault tolerance

B

It synchronizes optimizer states every microstep rather than once per step

C

It must all-gather parameters on-demand during both the forward and backward pass, on top of the reduce-scatter of gradients, adding an extra pass of parameter communication

D

It replaces all-reduce with an all-to-all, which has strictly higher bandwidth complexity

E

It uses tree all-reduce instead of ring all-reduce, which is bandwidth-suboptimal