M4-013M4: Sequences & TransformersTransformer Architecture AnatomyEasy
⚡Flashcard Drill➔Mastery:
Complete Structural Anatomy of a Modern Transformer Block
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
A modern Pre-LN Transformer block consists of RMSNorm, Multi-Head / Grouped-Query Attention, Residual Skip, RMSNorm, and a SwiGLU Feed-Forward Network.
📌 Key Takeaways
- •Sub-layer 1: Pre-RMSNorm Multi-Head Attention (MHA/GQA) with RoPE Residual Addition
- •Sub-layer 2: Pre-RMSNorm SwiGLU MLP Block Residual Addition
- •Modern standard: Pre-LN configuration guarantees an unobstructed gradient highway throughout hundreds of layers
📐 Mathematical Derivations
Mathematical Formulations of a Modern LLaMA-style Block:
Let input hidden representation be .
① Attention Sub-Layer:
.
Compute projections with Rotary Position Embeddings (RoPE):
.
Compute FlashAttention with causal mask:
.
Project back and add residual skip:
.
② Feed-Forward Sub-Layer (SwiGLU):
.
Compute gated activation with intermediate dimension :
.
Add second residual skip:
.
Let input hidden representation be .
① Attention Sub-Layer:
.
Compute projections with Rotary Position Embeddings (RoPE):
.
Compute FlashAttention with causal mask:
.
Project back and add residual skip:
.
② Feed-Forward Sub-Layer (SwiGLU):
.
Compute gated activation with intermediate dimension :
.
Add second residual skip:
.
🏭 Production Trade-offs
Post-LN vs Pre-LN Historical Note: Original Vaswani Transformer placed LayerNorm after residual addition (). Modern models universally place normalization before sub-layers (Pre-LN / Pre-RMSNorm) to guarantee gradient stability.
⚠️ Common Interview Pitfalls
- ✕Omitting the final LayerNorm/RMSNorm after the last Transformer block before the output unembedding head
- ✕Using standard ReLU in modern Transformer MLPs instead of SwiGLU or GELU
🎯 Interviewer Follow-up Inquiries
- ?Why does the residual connection must bypass the normalization layer in Pre-LN architectures?
- ?What architectural changes distinguish a Decoder block from an Encoder-Decoder cross-attention block?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚