M4-020M4: Sequences & TransformersTransformer Architecture AnatomyHard
⚡Flashcard Drill➔Mastery:
Transformer Weight Initialization Strategies and Maximal Update Parametrization (μP)
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Deep Transformers require scaled residual initializations () or zero-init output heads; μP coordinates layer width with learning rates to enable zero-shot hyperparameter transfer.
📌 Key Takeaways
- •GPT-2 Residual Scaling: scale output projection weights () by to bound accumulated residual variance
- •Zero-init heads (ReZero / adaLN-Zero): initialize final sub-layer projections to zero, starting training as pure identity maps
- •Maximal Update Parametrization (μP): scales hidden weights and output LR , allowing hyperparameter tuning on small models
📐 Mathematical Derivations
Mathematical Formulations:
① GPT-2 Residual Variance Scaling:
Each Transformer layer has two residual additions: .
If each residual branch has output variance , after layers the total variance grows as . To ensure , weights of projection layers are initialized with: .
② μP Rules for Transformers (Yang et al., 2022):
Under Standard Parametrization (SP), scaling width alters gradient dynamics, causing optimal learning rate to shift.
- Query projections (): Variance ; Learning rate .
- Key projections (): Variance ; Attention logits scaled by (instead of ).
- Value projections () and FFN: Variance ; Learning rate .
- Output Unembedding (): Variance ; Learning rate .
Under μP, feature representations adapt at identical rates regardless of width, allowing optimal hyperparameters tuned on a 1B model to transfer directly to 70B+.
① GPT-2 Residual Variance Scaling:
Each Transformer layer has two residual additions: .
If each residual branch has output variance , after layers the total variance grows as . To ensure , weights of projection layers are initialized with: .
② μP Rules for Transformers (Yang et al., 2022):
Under Standard Parametrization (SP), scaling width alters gradient dynamics, causing optimal learning rate to shift.
- Query projections (): Variance ; Learning rate .
- Key projections (): Variance ; Attention logits scaled by (instead of ).
- Value projections () and FFN: Variance ; Learning rate .
- Output Unembedding (): Variance ; Learning rate .
Under μP, feature representations adapt at identical rates regardless of width, allowing optimal hyperparameters tuned on a 1B model to transfer directly to 70B+.
🏭 Production Trade-offs
Cost economics: Tuning learning rate on 70B models costs hundreds of thousands of dollars in compute. μP allows teams to execute 100 trials on a 100M parameter model and apply the optimal settings to the full-scale pretraining run with zero re-tuning.
⚠️ Common Interview Pitfalls
- ✕Using standard PyTorch default initialization without residual scaling on deep Transformers (), causing early loss spikes
- ✕Applying μP rules to depth scaling; μP strictly governs width scaling (depth scaling requires distinct depth-μP formulations)
🎯 Interviewer Follow-up Inquiries
- ?Why must attention logits be scaled by rather than in the infinite-width μP limit?
- ?How does zero-initialization of output projections (ReZero) eliminate the need for learning rate warmup?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚