M4-020M4: Sequences & TransformersTransformer Architecture AnatomyHard
⚡Flashcard Drill➔Mastery:

Transformer Weight Initialization Strategies and Maximal Update Parametrization (μP)

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-021 →
📐 Mathematical Definition
std: σ∝1/d;μP: hidden 1/d, output 1/d, η∝1/d\text{std}:\ \sigma\propto1/\sqrt{d};\qquad \mu\text{P}:\ \text{hidden}\ 1/\sqrt{d},\ \text{output}\ 1/d,\ \eta\propto1/d
⚡ One-Sentence Core Answer
Deep Transformers require scaled residual initializations (1/2L1/\sqrt{2L}) or zero-init output heads; μP coordinates layer width with learning rates to enable zero-shot hyperparameter transfer.

📌 Key Takeaways

  • •
    GPT-2 Residual Scaling: scale output projection weights (WO,W2W_O, W_2) by 12L\frac{1}{\sqrt{2L}} to bound accumulated residual variance
  • •
    Zero-init heads (ReZero / adaLN-Zero): initialize final sub-layer projections to zero, starting training as pure identity maps
  • •
    Maximal Update Parametrization (μP): scales hidden weights ∝1/width\propto 1/\text{width} and output LR ∝1/width\propto 1/\text{width}, allowing hyperparameter tuning on small models

📐 Mathematical Derivations

Mathematical Formulations:
① GPT-2 Residual Variance Scaling:
Each Transformer layer has two residual additions: xl+1=xl+Attn(xl)+FFN(xl′)x_{l+1} = x_l + \text{Attn}(x_l) + \text{FFN}(x_l').
If each residual branch has output variance σ2\sigma^2, after LL layers the total variance grows as Var(xL)≈x0+2Lσ2\text{Var}(x_L) \approx x_0 + 2L \sigma^2. To ensure Var(xL)=O(1)\text{Var}(x_L) = O(1), weights of projection layers are initialized with: Std=0.022L\text{Std} = \frac{0.02}{\sqrt{2L}}.
② μP Rules for Transformers (Yang et al., 2022):
Under Standard Parametrization (SP), scaling width d→∞d \to \infty alters gradient dynamics, causing optimal learning rate η∗\eta^* to shift.
- Query projections (WQW_Q): Variance ∼1/d\sim 1/d; Learning rate ∼Θ(1)\sim \Theta(1).
- Key projections (WKW_K): Variance ∼1/d\sim 1/d; Attention logits scaled by 1/d1/d (instead of 1/d1/\sqrt{d}).
- Value projections (WVW_V) and FFN: Variance ∼1/d\sim 1/d; Learning rate ∼Θ(1)\sim \Theta(1).
- Output Unembedding (WoutW_{\text{out}}): Variance ∼1/d2\sim 1/d^2; Learning rate ∼Θ(1/d)\sim \Theta(1/d).
Under μP, feature representations adapt at identical rates regardless of width, allowing optimal hyperparameters tuned on a 1B model to transfer directly to 70B+.

🏭 Production Trade-offs

Cost economics: Tuning learning rate on 70B models costs hundreds of thousands of dollars in compute. μP allows teams to execute 100 trials on a 100M parameter model and apply the optimal settings to the full-scale pretraining run with zero re-tuning.
⚠️ Common Interview Pitfalls
  • ✕
    Using standard PyTorch default initialization without residual scaling on deep Transformers (L≥32L \ge 32), causing early loss spikes
  • ✕
    Applying μP rules to depth scaling; μP strictly governs width scaling (depth scaling requires distinct depth-μP formulations)
🎯 Interviewer Follow-up Inquiries
  • ?
    Why must attention logits be scaled by 1/d1/d rather than 1/d1/\sqrt{d} in the infinite-width μP limit?
  • ?
    How does zero-initialization of output projections (ReZero) eliminate the need for learning rate warmup?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-019: Normalization Architecture Choices in Transformers: Pre-LN, Post-LN, and RMSNorm📋 Back to BankNext →M4-021: The Residual Stream Perspective in Transformer Mechanistic Interpretability