1 / 8 Mixture-of-Experts

Pick Your Experts

In a Mixture-of-Experts transformer layer using top-k routing, how is a given token's output actually produced?

A

The token's hidden vector is split into k equal chunks, with one chunk sent to each of the k experts

B

A gating network scores all experts for the token, the top-k highest-scoring experts process it, and their outputs are combined in a weighted sum using the (normalized) gate scores

C

The gating network picks k experts uniformly at random each forward pass to force exploration

D

Every expert in the layer processes the token and their outputs are averaged with equal weight