GPipe to 1F1B, What Actually Changes

A 32-layer model is split across $p = 8$ pipeline stages and each batch is fed as $m = 64$ microbatches. The team switches the schedule from GPipe (all forwards, then all backwards) to PipeDream-Flush / 1F1B, keeping the same $p$ , the same $m$ , and the same layer partitioning.

Using Megatron-LM's accounting, what changes?

Both improve by about 8x, since the schedule shortens the bubble in the same proportion as the memory it saves.

Activation memory drops 64x, to a single microbatch, because each stage frees its activations as soon as it forwards them to the next stage.

The bubble fraction is unchanged at $(p-1)/m = 7/64$ , about 11%, but peak activation memory on the first stage drops roughly 8x: 1F1B stashes activations for at most $p = 8$ in-flight microbatches instead of GPipe's 64.

The bubble nearly disappears because forward and backward work now overlap, while activation memory is unchanged.

Neither changes: 1F1B only reorders gradient accumulation, which affects numerics rather than memory or utilization.