M4-022M4: Sequences & TransformersTransformer Architecture AnatomyMedium
⚡Flashcard Drill➔Mastery:
Depth vs Width Trade-offs in Transformers: Scaling Laws and Optimal Allocations
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Given a compute budget, scaling laws show depth and width should scale roughly in balance (), avoiding extreme wide-shallow or narrow-deep architectural extremes.
📌 Key Takeaways
- •Compute formula: FLOPs per token ; compute scales quadratically with width and linearly with depth
- •Balanced scaling: Kaplan et al. and Chinchilla demonstrated that optimal models scale depth and width proportionally
- •Inference latency constraint: deep models () suffer from high sequential memory roundtrips and latency during autoregressive token generation
📐 Mathematical Derivations
Scaling Dynamics and Complexity Analysis:
For a Transformer with depth , width , and intermediate dimension (or ):
- Parameters (excluding embeddings): .
- Training compute per token: FLOPs (2 forward + 4 backward).
- Total compute across tokens: .
Empirical Optimal Scaling Exponents:
When compute budget is increased, how should parameters be split between and ?
- If a model is made extremely deep and narrow (): Sequential inference latency per token scales linearly with , making real-time serving painfully slow. Gradient propagation and memory roundtrips degrade hardware efficiency.
- If a model is made extremely wide and shallow (): The model lacks compositional reasoning depth, and attention logit dynamic ranges become difficult to optimize.
- Industry Rule of Thumb: Hidden dimension to depth ratio typically ranges between to (e.g., LLaMA-7B: ; LLaMA-70B: ).
For a Transformer with depth , width , and intermediate dimension (or ):
- Parameters (excluding embeddings): .
- Training compute per token: FLOPs (2 forward + 4 backward).
- Total compute across tokens: .
Empirical Optimal Scaling Exponents:
When compute budget is increased, how should parameters be split between and ?
- If a model is made extremely deep and narrow (): Sequential inference latency per token scales linearly with , making real-time serving painfully slow. Gradient propagation and memory roundtrips degrade hardware efficiency.
- If a model is made extremely wide and shallow (): The model lacks compositional reasoning depth, and attention logit dynamic ranges become difficult to optimize.
- Industry Rule of Thumb: Hidden dimension to depth ratio typically ranges between to (e.g., LLaMA-7B: ; LLaMA-70B: ).
🏭 Production Trade-offs
Serving vs Training Trade-off: Deep models require sequential GPU kernel launches per generated token. Wider, moderately deep models achieve higher parallel Tensor Core utilization and lower TTFT (Time To First Token).
⚠️ Common Interview Pitfalls
- ✕Designing an excessively deep model () without considering the sequential GPU memory launch latency during production inference
- ✕Scaling width while keeping batch size tiny, causing GPU compute cores to underflow
🎯 Interviewer Follow-up Inquiries
- ?Why do wider models achieve higher hardware FLOP utilization (MFU) on modern GPUs than deeper models of equal parameter count?
- ?What is the Chinchilla optimal compute allocation between model parameter count and token count ?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚