Mixed Precision FP16/BF16/FP8

Mixed precision uses low-precision formats (FP16/BF16/FP8) for the forward and backward passes while keeping FP32 master weights and optimizer states, trading precision for speed and memory. Bit layouts and ranges: ① FP16 = 1 sign + 5 exponent + 10 mantissa; max value $65504 = 2^{15} \times (2 - 2^{-10})$; min normal $2^{-14} \approx 6.1\times10^{-5}$; min subnormal $2^{-24} \approx 6\times10^{-8}$. Problem: in deep networks / large-batch training gradients are often below $10^{-8}$ and underflow to zero in FP16 → parameters stop updating. Solution: loss scaling — multiply the loss by a factor $S$ (commonly $2^{24}$ or 1024) before backprop so gradients are amplified $S$×, then divide back on the FP32 master weights before updating; dynamic loss scaling: on Inf/NaN detection set $S \leftarrow S/2$ and skip the step; after a period without overflow, double $S$. ② BF16 = 1 sign + 8 exponent + 7 mantissa: the exponent width equals FP32's, so the dynamic range matches FP32 (about $\pm3.4\times10^{38}$), eliminating underflow entirely with no loss scaling needed; the cost is only 7 mantissa bits (~3 significant decimal digits). ③ FP8: E4M3 (4 exponent + 3 mantissa, max ~448; better precision, smaller range) for the forward pass (weights/activations); E5M2 (5 exponent + 2 mantissa, max ~57344; larger range, lower precision) for backward gradients (preventing overflow/underflow). Standard pipeline: FP32 master weights → copy a low-precision view each step for forward → multiply loss by $S$ and backprop → unscale the low-precision gradients and update FP32 master weights (EMA, weight decay, gradient clipping can all be composed in FP32).

standard for large-model training/inference (natively supported on A100/H100/TPU, tensor-core throughput doubling as precision halves); interview favorites: why FP16 needs loss scaling, BF16 vs FP16, and why FP8 splits into E4M3/E5M2.

trading precision for speed and memory — memory halves and Tensor Core throughput doubles (FP8 doubles it again on H100) without losing training quality: FP32 master weights preserve numerical precision; BF16 trades range for precision, fixing FP16's underflow (no loss scaling needed, more stable training); FP8 assigns range vs precision by role (forward activations need precision to avoid information loss, gradients need range to avoid overflow), the next-generation doubling for both inference and training throughput.

Test Your Knowledge: Practice Questions for "Mixed Precision FP16/BF16/FP8"

Single choice pitfall questions with instant feedback and mistake tracking.