Question 883

Capacity Overflow

An MoE layer sets expert capacity factor to 1.25 with 8 experts and a batch of 1024 tokens routed top-1. During a training step, one expert's assigned token count exceeds its capacity slot. What is the standard behavior for the overflow tokens?

A

The capacity factor is automatically raised mid-step to accommodate all tokens

B

They are dropped: skipped by that expert and passed through via the residual connection unprocessed (or routed to a fallback)

C

They are queued and processed in the next micro-batch to avoid any loss of information

D

They are forcibly split across two experts using a fractional gating weight