The Overflow Bucket

Capacity factor limits the total number of experts instantiated in the model, while the auxiliary loss penalizes the router for choosing different experts across consecutive layers

Capacity factor caps each expert's token budget per batch at roughly (tokens/num_experts) times the factor, with tokens beyond that dropped or passed through a residual; the auxiliary loss penalizes correlation between router probabilities and per-expert dispatch fractions to push routing toward balance during training

Capacity factor is a per-expert learned scalar that rescales that expert's output magnitude, while the auxiliary loss regularizes the expert weight matrices toward orthogonality

Capacity factor sets the number of experts activated per token (i.e. it is another name for k), while the auxiliary loss is a standard label-smoothing term applied to the final softmax