H5Medium★ Core Essential · Industrial BedrockPart H · Optimizers & Distributed Systems

PyTorch Automatic Mixed Precision (AMP) Workflow

⚡Flashcard Drill➔

Standard PyTorch AMP loop illustrating `autocast`, dynamic `GradScaler`, unscale-before-clip, and conditional step.

⏱️ Time: 显存吞吐翻倍,算力耗时减少 30%~50%
💾 Space: 激活显存减半
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
# PyTorch 工业级混合精度骨架(可在面试白板直接手撕的规范代码)
import torch
import torch.nn as nn
from torch.cuda.amp import autocast, GradScaler
def train_one_epoch_amp(model, dataloader, optimizer, criterion, device):
model.train()
scaler =
for batch_idx, (inputs, targets) in enumerate(dataloader):
inputs, targets = inputs.to(device), targets.to(device)
optimizer.zero_grad(set_to_none=True) # 设为 None 节省显存带宽
# 1. 混合精度前向计算 (自动在 Tensor Core 执行 FP16,在重要算子保留 FP32)
with autocast():
outputs = model(inputs)
loss =
# 2. 损失缩放并反向传播
scaler.scale(loss).backward()
# 3. 梯度裁剪前必须显式 Unscale
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
# 4. 优化器更新 (内部自动检查是否包含 NaN/Inf,若有则跳过更新)
scaler.step(optimizer)
# 5. 更新缩放因子 S
scaler.update()
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How do FP16 and BF16 differ at the bit level, and why do modern LLMs favor BF16?
Next: Ring All-Reduce Distributed Communication →
💡

Core Mental Anchor / Mnemonic

Run autocast forward, scale loss before backward, unscale before clipping, then step & update scaler.

📐 Mathematical Derivation & Core Formula

Lscaled=L⋅S,gunscaled=∇LscaledS,S←{S⋅2若连续 N 步无 NaN/InfS/2若检测到溢出\mathcal{L}_{\text{scaled}} = \mathcal{L} \cdot S, \quad g_{\text{unscaled}} = \frac{\nabla \mathcal{L}_{\text{scaled}}}{S}, \quad S \leftarrow \begin{cases} S \cdot 2 & \text{若连续 } N \text{ 步无 NaN/Inf} \\ S / 2 & \text{若检测到溢出} \end{cases}
### FP16 Gradient Underflow & Dynamic GradScaler
FP16 only has 5 exponent bits (minimum positive normal ≈6.1×10−5\approx 6.1 \times 10^{-5}, subnormal ≈6×10−8\approx 6 \times 10^{-8}), causing small backprop gradients to flush to zero.
GradScaler multiplies loss by a large factor SS (e.g., 2162^{16}) before .backward() to keep gradients above underflow, then unscales by 1/S1/S before optimizer.step(). If any inf/nan is detected, the optimizer step is skipped and SS is halved; after stable runs (e.g., 2000 steps), SS doubles.

🔄 Tensor Dimensions & Shape Flow

autocast forward -> scaler.scale(loss).backward() -> scaler.unscale_(opt) -> clip_grad_norm_ -> scaler.step -> update

🛡️ Numerical Stability & Pitfalls Checklist

  • Use optimizer.zero_grad(set_to_none=True) to skip zero-filling tensor memory.
  • Always call scaler.unscale_(optimizer) before clip_grad_norm_ so norms match unscaled gradients.
  • BF16 matches FP32 dynamic exponent range and generally does not require GradScaler.

💻 Reference Code Implementation

# PyTorch 工业级混合精度骨架(可在面试白板直接手撕的规范代码)
import torch
import torch.nn as nn
from torch.cuda.amp import autocast, GradScaler

def train_one_epoch_amp(model, dataloader, optimizer, criterion, device):
    model.train()
    scaler = GradScaler() # 初始化动态梯度缩放器
    
    for batch_idx, (inputs, targets) in enumerate(dataloader):
        inputs, targets = inputs.to(device), targets.to(device)
        optimizer.zero_grad(set_to_none=True) # 设为 None 节省显存带宽
        
        # 1. 混合精度前向计算 (自动在 Tensor Core 执行 FP16,在重要算子保留 FP32)
        with autocast():
            outputs = model(inputs)
            loss = criterion(outputs, targets)
            
        # 2. 损失缩放并反向传播
        scaler.scale(loss).backward()
        
        # 3. 梯度裁剪前必须显式 Unscale
        scaler.unscale_(optimizer)
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        
        # 4. 优化器更新 (内部自动检查是否包含 NaN/Inf,若有则跳过更新)
        scaler.step(optimizer)
        
        # 5. 更新缩放因子 S
        scaler.update()

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

# 概念验证代码
assert hasattr(autocast, '__enter__'), "autocast 必须作为上下文管理器"
print("✓ PyTorch 混合精度训练骨架结构校验通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How do FP16 and BF16 differ at the bit level, and why do modern LLMs favor BF16?
FP16 uses 1 sign + 5 exponent + 10 mantissa bits (narrow dynamic range, needs GradScaler). BF16 uses 1 sign + 8 exponent + 7 mantissa bits—matching FP32 exponent range (∼3.4×1038\sim 3.4 \times 10^{38})—eliminating overflow/underflow scaling complexity during LLM pretraining.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Exponential Moving Average (EMA) of WeightsBack to Part H & All 105Next: Ring All-Reduce Distributed Communication →