Question 739

Same Range, Blurrier Picture

bf16 can often replace fp32 in deep learning training and inference with minimal risk of overflow, despite using half the storage. Why?

A

bf16 uses a wider exponent field than fp32, giving it extra headroom above fp32's range

B

bf16 doubles the mantissa bit count relative to fp32 while shrinking the exponent

C

bf16 uses the same 8-bit exponent as fp32, giving it the same dynamic range but fewer mantissa bits (7 vs 23), so it loses precision, not range

D

bf16 automatically rescales every value into a normalized window before each operation

E

bf16 stores values as fixed-point integers, which cannot overflow