Bias, Not Backprop

The bias term is added directly to the gate weight that scales each selected expert's output before the weighted sum, so an overloaded expert's contribution to the hidden state is shrunk in proportion to its load.

The per-expert bias is a learned parameter optimized end-to-end via backpropagation through the routing softmax, jointly with the rest of the model's weights.

A per-expert bias term is added to the routing affinity score used only for top-k expert selection (never to the gate weight that scales the expert's output), and after each training step the bias is nudged up for underloaded experts and down for overloaded ones by a fixed step size; a tiny complementary sequence-wise balance loss is kept only as a backstop against extreme per-sequence imbalance.

A large auxiliary cross-entropy-style loss term is added directly to the language modeling loss to penalize deviation from uniform expert usage across the batch, essentially the same mechanism used in Switch Transformer and GShard.

Balance is enforced purely through capacity-based token dropping: tokens routed to an already-full expert are silently discarded, with no bias or gradient signal involved.