Block-Scaled FP8

DeepSeek-V3's FP8 training recipe scales activations per 1x128 tile and weights per 128x128 block, and uses E4M3 (4 exponent bits, 3 mantissa bits) for essentially all FP8 GEMMs, departing from the classic mixed-precision convention of E4M3 for the forward pass and E5M2 (wider exponent range, fewer mantissa bits) for the gradient-heavy backward pass. Why does the fine-grained block/tile scaling make it safe to drop E5M2's extra exponent range even for backward-pass GEMMs?

DeepSeek-V3 never runs gradient-producing GEMMs in FP8 at all, only forward-pass activations, so E5M2's wider range was never actually needed in the first place.

Each small tile/block gets its own independently fit scale factor, so outliers are localized within a group instead of forcing one tensor-wide scale to span both extreme and typical values, so every group's rescaled values fit inside E4M3's narrower representable range without needing E5M2's wider exponent field.

Hopper-generation tensor cores only implement native multiply-accumulate for E4M3, not E5M2, so the format choice was forced by hardware regardless of the dynamic-range tradeoff.

Fine-grained block scaling raises E4M3's effective mantissa precision by averaging rounding error across each block, which compensates for its coarser exponent range.