M4-015M4: Sequences & TransformersTransformer Architecture AnatomyMedium
⚡Flashcard Drill➔Mastery:
Role of the Feed-Forward Network (FFN) and the 4x / (8/3)x Hidden Dimension Rationale
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
FFN layers act as associative key-value memory storing factual knowledge; standard MLPs expand hidden dimension by , while SwiGLU uses to preserve parameter parity.
📌 Key Takeaways
- •Knowledge storage: Geva et al. demonstrated FFN layers store factual associations, while attention routes information
- •Channel mixing: provides position-wise non-linear transformations across feature channels ()
- •Parameter parity: SwiGLU introduces a third weight matrix; scaling intermediate dimension to keeps FLOPs and parameter count identical to standard MLPs
📐 Mathematical Derivations
Mathematical Formulations and Parameter Parity:
① Standard 2-Layer MLP (ReLU/GELU):
, where and .
Standard expansion sets .
Parameter count: parameters per block.
② Gated SwiGLU MLP (LLaMA/Mistral):
.
Notice there are three weight matrices instead of two: .
Total parameters: .
To maintain exact parameter parity with the standard budget:
.
(In LLaMA, , rounded to a multiple of 256 for optimal GPU Tensor Core tile alignment).
① Standard 2-Layer MLP (ReLU/GELU):
, where and .
Standard expansion sets .
Parameter count: parameters per block.
② Gated SwiGLU MLP (LLaMA/Mistral):
.
Notice there are three weight matrices instead of two: .
Total parameters: .
To maintain exact parameter parity with the standard budget:
.
(In LLaMA, , rounded to a multiple of 256 for optimal GPU Tensor Core tile alignment).
🏭 Production Trade-offs
Parameter distribution in Transformers: FFN layers contain approximately of all non-embedding parameters in a Transformer, while multi-head attention contains . FFNs dominate memory capacity, while attention dominates sequence context routing.
⚠️ Common Interview Pitfalls
- ✕Adopting SwiGLU with without adjusting for the 50% parameter and FLOP inflation
- ✕Setting intermediate hidden dimensions to odd numbers not divisible by 64 or 128, severely degrading GPU Tensor Core memory coalescing
🎯 Interviewer Follow-up Inquiries
- ?How do FFN layers function as key-value associative memories in Transformer interpretability research?
- ?Why is the intermediate dimension in LLaMA rounded to the nearest multiple of 256?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚