2 / 9 Quantization & Low Precision

Half a Byte, Two Flavors

NVIDIA's Transformer Engine supports two 8-bit floating point formats, E4M3 and E5M2. What's the core tradeoff between them?

A

E4M3 is a 4-bit format and E5M2 is an 8-bit format, so they aren't directly comparable

B

E4M3 is reserved for integer quantization and cannot represent floating point values

C

E4M3 (4 exponent, 3 mantissa bits) favors precision over range and suits weights/activations; E5M2 (5 exponent, 2 mantissa bits) favors range over precision and suits gradients with larger dynamic swings

D

E4M3 and E5M2 have identical range and precision and differ only in naming convention

E

E5M2 always gives better accuracy than E4M3 because more exponent bits means more accuracy