Four-Way Split, Zero Bubble

DeepSeek-V3's DualPipe schedule splits each pipeline micro-batch chunk into four components (attention, all-to-all dispatch, MLP, all-to-all combine) and runs forward and backward micro-batches from opposite ends of the pipeline concurrently, reserving a fixed share of SMs for communication. Cross-node expert-parallel MoE layers require an all-to-all dispatch and combine per layer. What specific problem does DualPipe's bidirectional, component-level interleaving solve that a standard 1F1B (one-forward-one-backward) schedule does not solve on its own?

It removes the need for expert parallelism entirely by letting every GPU hold every expert's parameters locally.

It eliminates gradient synchronization across data-parallel replicas because gradients are already computed twice, once per pipeline direction.

1F1B interleaves compute across different micro-batches but doesn't by itself hide the extra cross-node all-to-all traffic MoE layers add; DualPipe deliberately pairs each chunk's all-to-all with concurrent compute from an oppositely-directed chunk so the all-to-all latency is fully overlapped rather than sitting on the critical path as extra bubble.

It halves the number of pipeline stages required by running forward and backward simultaneously, which reduces per-GPU parameter memory.

It converts the all-to-all collective into asynchronous point-to-point sends that require no GPU streaming multiprocessors at all.