H4EasyPart H · Optimizers & Distributed Systems
Exponential Moving Average (EMA) of Weights
Track exponential moving average shadow parameters for smoother loss basins and improved evaluation.
⏱️ Time:
O(P)💾 Space:
O(P) 需要额外常驻一份权重显存🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class ModelEMA:
def __init__(self, model_params: list, decay: float = 0.999):
self.decay = decay
# 影子权重深拷贝初始化
self.shadow_params =
def update(self, current_params: list):
for s, c in zip(self.shadow_params, current_params):
# s = decay * s + (1 - decay) * c
s *= self.decay
s += (1.0 - self.decay) * c
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does Stochastic Weight Averaging (SWA) differ from EMA?
💡
Core Mental Anchor / Mnemonic
Maintain decoupled shadow weights updated smoothly via exponential decay at each optimizer step.
📐 Mathematical Derivation & Core Formula
### Temporal Weight Averaging in Parameter Space
EMA maintains shadow weights updated each step as with .
Averaging recent optimization iterates dampens SGD/Adam noise around the optimum, steering inference weights toward flatter minima with better generalization (commonly used in diffusion models and vision backbones).
EMA maintains shadow weights updated each step as with .
Averaging recent optimization iterates dampens SGD/Adam noise around the optimum, steering inference weights toward flatter minima with better generalization (commonly used in diffusion models and vision backbones).
🔄 Tensor Dimensions & Shape Flow
model params -> shadow = decay * shadow + (1 - decay) * param
🛡️ Numerical Stability & Pitfalls Checklist
- Warm up
decaygradually from smaller values (e.g. 0.9) toward 0.999–0.9999 early in training. - Execute EMA updates inside
no_gradso shadow weights never join the autograd graph.
💻 Reference Code Implementation
import numpy as np
class ModelEMA:
def __init__(self, model_params: list, decay: float = 0.999):
self.decay = decay
# 影子权重深拷贝初始化
self.shadow_params = [p.copy() for p in model_params]
def update(self, current_params: list):
for s, c in zip(self.shadow_params, current_params):
# s = decay * s + (1 - decay) * c
s *= self.decay
s += (1.0 - self.decay) * c
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
w = [np.array([10.0])]
ema = ModelEMA(w, decay=0.9)
# 更新当前权重变为 0.0
ema.update([np.array([0.0])])
# 影子权重应为 0.9 * 10 + 0.1 * 0 = 9.0
assert np.isclose(ema.shadow_params[0][0], 9.0)
print("✓ 权重 EMA 自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How does Stochastic Weight Averaging (SWA) differ from EMA?
EMA maintains a continuous exponentially decayed moving average at every step, while SWA computes an equal-weight arithmetic average of discrete model snapshots collected near the end of training or cyclical LR epochs.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.