Mixed Precision FP16/BF16/FP8

Mixed precision uses low-precision formats (FP16/BF16/FP8) for the forward and backward passes while keeping FP32 master weights and optimizer states, trading precision for speed and memory. Bit layouts and ranges: ① FP16 = 1 sign + 5 exponent + 10 mantissa; max value $65504 = 2^{15} \times (2 - 2^{-10})$; min normal $2^{-14} \approx 6.1\times10^{-5}$; min subnormal $2^{-24} \approx 6\times10^{-8}$. Problem: in deep networks / large-batch training gradients are often below $10^{-8}$ and underflow to zero in FP16 → parameters stop updating. Solution: loss scaling — multiply the loss by a factor $S$ (commonly $2^{24}$ or 1024) before backprop so gradients are amplified $S$×, then divide back on the FP32 master weights before updating; dynamic loss scaling: on Inf/NaN detection set $S \leftarrow S/2$ and skip the step; after a period without overflow, double $S$. ② BF16 = 1 sign + 8 exponent + 7 mantissa: the exponent width equals FP32's, so the dynamic range matches FP32 (about $\pm3.4\times10^{38}$), eliminating underflow entirely with no loss scaling needed; the cost is only 7 mantissa bits (~3 significant decimal digits). ③ FP8: E4M3 (4 exponent + 3 mantissa, max ~448; better precision, smaller range) for the forward pass (weights/activations); E5M2 (5 exponent + 2 mantissa, max ~57344; larger range, lower precision) for backward gradients (preventing overflow/underflow). Standard pipeline: FP32 master weights → copy a low-precision view each step for forward → multiply loss by $S$ and backprop → unscale the low-precision gradients and update FP32 master weights (EMA, weight decay, gradient clipping can all be composed in FP32).

standard for large-model training/inference (natively supported on A100/H100/TPU, tensor-core throughput doubling as precision halves); interview favorites: why FP16 needs loss scaling, BF16 vs FP16, and why FP8 splits into E4M3/E5M2.

trading precision for speed and memory — memory halves and Tensor Core throughput doubles (FP8 doubles it again on H100) without losing training quality: FP32 master weights preserve numerical precision; BF16 trades range for precision, fixing FP16's underflow (no loss scaling needed, more stable training); FP8 assigns range vs precision by role (forward activations need precision to avoid information loss, gradients need range to avoid overflow), the next-generation doubling for both inference and training throughput.

Write FP16's bit layout (1 sign + 5 exponent + 10 mantissa) and max value $65504 = 2^{15} \times (2 - 2^{-10})$; explain why training gradients often underflow (below $2^{-24} \approx 6\times10^{-8}$).

Loss scaling: why multiply the loss by $S$ before backprop and divide back on FP32 master weights before updating? Why does the dynamic policy (halve $S$ on overflow and skip the step, double after a clean streak) work?

Why doesn't BF16 (1+8+7) need loss scaling? Its exponent width equals FP32's, giving a range of about $\pm3.4\times10^{38}$; what is the cost (7 mantissa bits) and when is the precision insufficient?

Why are FP8 E4M3 and E5M2 used for the forward pass and gradients respectively? Compare their range/precision trade-offs (max ~448 vs ~57344, 3 vs 2 mantissa bits).

Standard mixed-precision pipeline: FP32 master weights, low-precision forward/backward, loss scaling and unscaling, FP32 updates; how does it compose with gradient clipping, EMA and DDP?