M3-011M3: Deep Learning FoundationsWeight InitializationMedium
⚡Flashcard Drill➔Mastery:

Common Initialization Techniques in Transformers

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-012 →
📐 Mathematical Definition
residual scale=12L\text{residual scale}=\frac{1}{\sqrt{2L}}
⚡ One-Sentence Core Answer
Key techniques include residual branch scaling (1/2L1/\sqrt{2L}), zero-initializing output projections, embedding rescaling, and small initial LayerNorm gains.

📌 Key Takeaways

  • •
    Residual scaling: scale attention and FFN output projections by 1/2L1/\sqrt{2L} to keep variance O(1)O(1) across LL layers
  • •
    Zero output projection: initialize WoutW_{\text{out}} and WdownW_{\text{down}} to 0, starting the network as a pure identity map (ReZero/Fixup)
  • •
    Embedding scaling: scale token embeddings by d\sqrt{d} (Vaswani et al.) or 1/d1/\sqrt{d} to match initial LayerNorm variances

📐 Mathematical Derivations

Motivations and Formulations:
① Residual Scaling ($1/\sqrt{2L}$) in GPT-2:
In a Transformer, the residual state accumulates as xl=xl−1+fattn(xl−1)+fffn(xl−1′)x_{l} = x_{l-1} + f_{\text{attn}}(x_{l-1}) + f_{\text{ffn}}(x'_{l-1}). Each layer adds two residual branches. If each branch contributes variance σ2\sigma^2, after LL layers the total variance grows linearly: Var(xL)≈x0+2Lσ2\text{Var}(x_L) \approx x_0 + 2L \sigma^2. To maintain Var(xL)=O(1)\text{Var}(x_L) = O(1), weights of output projections (WoW_o in attention, W2W_2 in MLP) are initialized with standard deviation scaled by 12L\frac{1}{\sqrt{2L}}.
② Zero-Initialization of Projections (ReZero / adaLN-Zero):
By initializing the final linear projection of each residual branch to zero, F(x)≡0F(x) \equiv 0 at step 0. The block behaves as an exact identity function y=xy = x. Gradients flow unimpeded through the skip connection highway, eliminating initial instability without requiring learning rate warm-up.

🏭 Production Trade-offs

Practical adoption: Deep Transformers (L≥32L \ge 32) without residual branch scaling or Pre-LN suffer from extreme gradient variance in early iterations, necessitating extensive learning rate warmup schedules (thousands of steps) to avoid immediate divergence.
⚠️ Common Interview Pitfalls
  • ✕
    Using standard N(0,1)\mathcal{N}(0, 1) or unscaled Kaiming initialization on Transformer linear projections, causing immediate training explosion
  • ✕
    Initializing input embeddings and output unembedding matrices with arbitrary random scales without checking scale compatibility with LayerNorm
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does initializing the output projection of a residual block to zero guarantee an exact identity transformation at step 0?
  • ?
    How does adaLN-Zero in Diffusion Transformers (DiT) implement zero-initialization on adaptive conditioning layers?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-010: The 'Variance Preservation' Principle and Its Relation to Gradient Stability📋 Back to BankNext →M3-012: Maximal Update Parametrization (μP): Concepts and Hyperparameter Transfer