Mixture-of-Experts

Top-k routing, all-to-all dispatch and combine, load balancing and capacity factors.

0 / 8 solved

Question Tags Difficulty Success

1. Pick Your Experts

moerouting +1

Medium —

2. Capacity Overflow

moetoken-dropping +1

Hard —

3. Traffic Jam at Expert Seven

moeexpert-parallelism +2

Hard —

4. Z-Loss Guardrail

moerouter +1

Hard —

5. The Overflow Bucket

moecapacity-factor +1

Insane —

6. All-to-All Wall

moeexpert-parallelism +1

Insane —

7. Bias, Not Backprop

moeload-balancing

Insane —

8. Sparse Compute, Dense Memory

moeexpert-parallelism +1

Insane —

Learning Resources

How to Scale Your Model (the scaling book)

Up next

8 of 8 questions here are still unsolved.

Next module Frontier & Recent Research