M3-011M3: Deep Learning FoundationsWeight InitializationMedium
⚡Flashcard Drill➔Mastery:
Common Initialization Techniques in Transformers
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Key techniques include residual branch scaling (), zero-initializing output projections, embedding rescaling, and small initial LayerNorm gains.
📌 Key Takeaways
- •Residual scaling: scale attention and FFN output projections by to keep variance across layers
- •Zero output projection: initialize and to 0, starting the network as a pure identity map (ReZero/Fixup)
- •Embedding scaling: scale token embeddings by (Vaswani et al.) or to match initial LayerNorm variances
📐 Mathematical Derivations
Motivations and Formulations:
① Residual Scaling ($1/\sqrt{2L}$) in GPT-2:
In a Transformer, the residual state accumulates as . Each layer adds two residual branches. If each branch contributes variance , after layers the total variance grows linearly: . To maintain , weights of output projections ( in attention, in MLP) are initialized with standard deviation scaled by .
② Zero-Initialization of Projections (ReZero / adaLN-Zero):
By initializing the final linear projection of each residual branch to zero, at step 0. The block behaves as an exact identity function . Gradients flow unimpeded through the skip connection highway, eliminating initial instability without requiring learning rate warm-up.
① Residual Scaling ($1/\sqrt{2L}$) in GPT-2:
In a Transformer, the residual state accumulates as . Each layer adds two residual branches. If each branch contributes variance , after layers the total variance grows linearly: . To maintain , weights of output projections ( in attention, in MLP) are initialized with standard deviation scaled by .
② Zero-Initialization of Projections (ReZero / adaLN-Zero):
By initializing the final linear projection of each residual branch to zero, at step 0. The block behaves as an exact identity function . Gradients flow unimpeded through the skip connection highway, eliminating initial instability without requiring learning rate warm-up.
🏭 Production Trade-offs
Practical adoption: Deep Transformers () without residual branch scaling or Pre-LN suffer from extreme gradient variance in early iterations, necessitating extensive learning rate warmup schedules (thousands of steps) to avoid immediate divergence.
⚠️ Common Interview Pitfalls
- ✕Using standard or unscaled Kaiming initialization on Transformer linear projections, causing immediate training explosion
- ✕Initializing input embeddings and output unembedding matrices with arbitrary random scales without checking scale compatibility with LayerNorm
🎯 Interviewer Follow-up Inquiries
- ?Why does initializing the output projection of a residual block to zero guarantee an exact identity transformation at step 0?
- ?How does adaLN-Zero in Diffusion Transformers (DiT) implement zero-initialization on adaptive conditioning layers?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚