8 / 8 Mixture-of-Experts

Sparse Compute, Dense Memory

DeepSeek-V3 activates roughly 37B of its 671B total parameters per token, routing each token to a handful of fine-grained experts plus a few always-on shared experts. Which statement best explains why this sparsity does not proportionally shrink the resources needed to train or serve the model at cluster scale?

A

Fine-grained expert segmentation reduces the total parameter count relative to a coarse-expert MoE of equal FLOPs, so any memory savings from sparse activation are simply cancelled out by the segmentation itself.

B

Experts not selected for a given token can simply be paged out to host memory or disk between forward passes at effectively zero cost, so the only real overhead is computing the top-k gate itself.

C

Per-token FLOPs scale with only the ~37B active parameters, but which experts fire is data-dependent per token, so all 671B parameters across all experts (sharded across devices) must stay resident in accelerator memory regardless; dispatching tokens to their assigned experts also requires an all-to-all exchange of actual hidden-state activations across devices, whose bandwidth cost scales with batch size and hidden dim rather than with sparsity, making communication rather than compute the binding constraint.

D

Inference always falls back to dense computation over every expert even though training uses sparse routing, so the 37B active-parameter figure is a training-only artifact that doesn't apply when serving the model.

E

The all-to-all exchange in expert parallelism only ships lightweight routing metadata such as expert indices between devices, so communication overhead is negligible and the true cost is idle experts occupying memory.