H5Medium★ Core Essential · Industrial BedrockPart H · Optimizers & Distributed Systems
PyTorch Automatic Mixed Precision (AMP) Workflow
Standard PyTorch AMP loop illustrating `autocast`, dynamic `GradScaler`, unscale-before-clip, and conditional step.
⏱️ Time:
显存吞吐翻倍,算力耗时减少 30%~50%💾 Space:
激活显存减半🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
# PyTorch 工业级混合精度骨架(可在面试白板直接手撕的规范代码)
import torch
import torch.nn as nn
from torch.cuda.amp import autocast, GradScaler
def train_one_epoch_amp(model, dataloader, optimizer, criterion, device):
model.train()
scaler =
for batch_idx, (inputs, targets) in enumerate(dataloader):
inputs, targets = inputs.to(device), targets.to(device)
optimizer.zero_grad(set_to_none=True) # 设为 None 节省显存带宽
# 1. 混合精度前向计算 (自动在 Tensor Core 执行 FP16,在重要算子保留 FP32)
with autocast():
outputs = model(inputs)
loss =
# 2. 损失缩放并反向传播
scaler.scale(loss).backward()
# 3. 梯度裁剪前必须显式 Unscale
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
# 4. 优化器更新 (内部自动检查是否包含 NaN/Inf,若有则跳过更新)
scaler.step(optimizer)
# 5. 更新缩放因子 S
scaler.update()
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How do FP16 and BF16 differ at the bit level, and why do modern LLMs favor BF16?
💡
Core Mental Anchor / Mnemonic
Run autocast forward, scale loss before backward, unscale before clipping, then step & update scaler.
📐 Mathematical Derivation & Core Formula
### FP16 Gradient Underflow & Dynamic GradScaler
FP16 only has 5 exponent bits (minimum positive normal , subnormal ), causing small backprop gradients to flush to zero.
FP16 only has 5 exponent bits (minimum positive normal , subnormal ), causing small backprop gradients to flush to zero.
GradScaler multiplies loss by a large factor (e.g., ) before .backward() to keep gradients above underflow, then unscales by before optimizer.step(). If any inf/nan is detected, the optimizer step is skipped and is halved; after stable runs (e.g., 2000 steps), doubles.🔄 Tensor Dimensions & Shape Flow
autocast forward -> scaler.scale(loss).backward() -> scaler.unscale_(opt) -> clip_grad_norm_ -> scaler.step -> update
🛡️ Numerical Stability & Pitfalls Checklist
- Use
optimizer.zero_grad(set_to_none=True)to skip zero-filling tensor memory. - Always call
scaler.unscale_(optimizer)beforeclip_grad_norm_so norms match unscaled gradients. - BF16 matches FP32 dynamic exponent range and generally does not require
GradScaler.
💻 Reference Code Implementation
# PyTorch 工业级混合精度骨架(可在面试白板直接手撕的规范代码)
import torch
import torch.nn as nn
from torch.cuda.amp import autocast, GradScaler
def train_one_epoch_amp(model, dataloader, optimizer, criterion, device):
model.train()
scaler = GradScaler() # 初始化动态梯度缩放器
for batch_idx, (inputs, targets) in enumerate(dataloader):
inputs, targets = inputs.to(device), targets.to(device)
optimizer.zero_grad(set_to_none=True) # 设为 None 节省显存带宽
# 1. 混合精度前向计算 (自动在 Tensor Core 执行 FP16,在重要算子保留 FP32)
with autocast():
outputs = model(inputs)
loss = criterion(outputs, targets)
# 2. 损失缩放并反向传播
scaler.scale(loss).backward()
# 3. 梯度裁剪前必须显式 Unscale
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
# 4. 优化器更新 (内部自动检查是否包含 NaN/Inf,若有则跳过更新)
scaler.step(optimizer)
# 5. 更新缩放因子 S
scaler.update()
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
# 概念验证代码
assert hasattr(autocast, '__enter__'), "autocast 必须作为上下文管理器"
print("✓ PyTorch 混合精度训练骨架结构校验通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How do FP16 and BF16 differ at the bit level, and why do modern LLMs favor BF16?
FP16 uses 1 sign + 5 exponent + 10 mantissa bits (narrow dynamic range, needs GradScaler). BF16 uses 1 sign + 8 exponent + 7 mantissa bits—matching FP32 exponent range ()—eliminating overflow/underflow scaling complexity during LLM pretraining.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.