H4EasyPart H · Optimizers & Distributed Systems

Exponential Moving Average (EMA) of Weights

⚡Flashcard Drill➔

Track exponential moving average shadow parameters for smoother loss basins and improved evaluation.

⏱️ Time: O(P)
💾 Space: O(P) 需要额外常驻一份权重显存
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class ModelEMA:
def __init__(self, model_params: list, decay: float = 0.999):
self.decay = decay
# 影子权重深拷贝初始化
self.shadow_params =
def update(self, current_params: list):
for s, c in zip(self.shadow_params, current_params):
# s = decay * s + (1 - decay) * c
s *= self.decay
s += (1.0 - self.decay) * c
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does Stochastic Weight Averaging (SWA) differ from EMA?
Next: PyTorch Automatic Mixed Precision (AMP) Workflow →
💡

Core Mental Anchor / Mnemonic

Maintain decoupled shadow weights updated smoothly via exponential decay at each optimizer step.

📐 Mathematical Derivation & Core Formula

θEMA←βθEMA+(1−β)θtrain\theta_{\text{EMA}} \leftarrow \beta \theta_{\text{EMA}} + (1 - \beta) \theta_{\text{train}}
### Temporal Weight Averaging in Parameter Space
EMA maintains shadow weights updated each step as θEMA(t)=βθEMA(t−1)+(1−β)θ(t)\theta_{\text{EMA}}^{(t)} = \beta \theta_{\text{EMA}}^{(t-1)} + (1 - \beta)\theta^{(t)} with β∈[0.999,0.9999]\beta \in [0.999, 0.9999].
Averaging recent optimization iterates dampens SGD/Adam noise around the optimum, steering inference weights toward flatter minima with better generalization (commonly used in diffusion models and vision backbones).

🔄 Tensor Dimensions & Shape Flow

model params -> shadow = decay * shadow + (1 - decay) * param

🛡️ Numerical Stability & Pitfalls Checklist

  • Warm up decay gradually from smaller values (e.g. 0.9) toward 0.999–0.9999 early in training.
  • Execute EMA updates inside no_grad so shadow weights never join the autograd graph.

💻 Reference Code Implementation

import numpy as np

class ModelEMA:
    def __init__(self, model_params: list, decay: float = 0.999):
        self.decay = decay
        # 影子权重深拷贝初始化
        self.shadow_params = [p.copy() for p in model_params]
        
    def update(self, current_params: list):
        for s, c in zip(self.shadow_params, current_params):
            # s = decay * s + (1 - decay) * c
            s *= self.decay
            s += (1.0 - self.decay) * c

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
w = [np.array([10.0])]
ema = ModelEMA(w, decay=0.9)
# 更新当前权重变为 0.0
ema.update([np.array([0.0])])
# 影子权重应为 0.9 * 10 + 0.1 * 0 = 9.0
assert np.isclose(ema.shadow_params[0][0], 9.0)
print("✓ 权重 EMA 自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How does Stochastic Weight Averaging (SWA) differ from EMA?
EMA maintains a continuous exponentially decayed moving average at every step, while SWA computes an equal-weight arithmetic average of discrete model snapshots collected near the end of training or cyclical LR epochs.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Gradient Clipping by Global NormBack to Part H & All 105Next: PyTorch Automatic Mixed Precision (AMP) Workflow →