Sparse Compute, Dense Memory

DeepSeek-V3 activates roughly 37B of its 671B total parameters per token, routing each token to a handful of fine-grained experts plus a few always-on shared experts. Which statement best explains why this sparsity does not proportionally shrink the resources needed to train or serve the model at cluster scale?

Fine-grained expert segmentation reduces the total parameter count relative to a coarse-expert MoE of equal FLOPs, so any memory savings from sparse activation are simply cancelled out by the segmentation itself.

Experts not selected for a given token can simply be paged out to host memory or disk between forward passes at effectively zero cost, so the only real overhead is computing the top-k gate itself.

Per-token FLOPs scale with only the ~37B active parameters, but which experts fire is data-dependent per token, so all 671B parameters across all experts (sharded across devices) must stay resident in accelerator memory regardless; dispatching tokens to their assigned experts also requires an all-to-all exchange of actual hidden-state activations across devices, whose bandwidth cost scales with batch size and hidden dim rather than with sparsity, making communication rather than compute the binding constraint.

Inference always falls back to dense computation over every expert even though training uses sparse routing, so the 37B active-parameter figure is a training-only artifact that doesn't apply when serving the model.

The all-to-all exchange in expert parallelism only ships lightweight routing metadata such as expert indices between devices, so communication overhead is negligible and the true cost is idle experts occupying memory.