M4-013M4: Sequences & TransformersTransformer Architecture AnatomyEasy
⚡Flashcard Drill➔Mastery:

Complete Structural Anatomy of a Modern Transformer Block

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-014 →
📐 Mathematical Definition
Block(x)=x+FFN(LN(x+MHA(LN(x))))  (Pre-LN)\text{Block}(x)=x+\mathrm{FFN}(\mathrm{LN}(x+\mathrm{MHA}(\mathrm{LN}(x))))\ \ (\text{Pre-LN})
⚡ One-Sentence Core Answer
A modern Pre-LN Transformer block consists of RMSNorm, Multi-Head / Grouped-Query Attention, Residual Skip, RMSNorm, and a SwiGLU Feed-Forward Network.

📌 Key Takeaways

  • •
    Sub-layer 1: Pre-RMSNorm →\to Multi-Head Attention (MHA/GQA) with RoPE →\to Residual Addition
  • •
    Sub-layer 2: Pre-RMSNorm →\to SwiGLU MLP Block →\to Residual Addition
  • •
    Modern standard: Pre-LN configuration guarantees an unobstructed gradient highway throughout hundreds of layers

📐 Mathematical Derivations

Mathematical Formulations of a Modern LLaMA-style Block:
Let input hidden representation be x∈RB×S×dx \in \mathbb{R}^{B \times S \times d}.
① Attention Sub-Layer:
xnorm1=RMSNorm(x)x_{\text{norm1}} = \text{RMSNorm}(x).
Compute projections with Rotary Position Embeddings (RoPE):
Q=RoPE(xnorm1WQ),  K=RoPE(xnorm1WK),  V=xnorm1WVQ = \text{RoPE}(x_{\text{norm1}} W_Q), \; K = \text{RoPE}(x_{\text{norm1}} W_K), \; V = x_{\text{norm1}} W_V.
Compute FlashAttention with causal mask:
AttnOut=Softmax(QKTdk+Mcausal)V\text{AttnOut} = \text{Softmax}\left( \frac{Q K^T}{\sqrt{d_k}} + M_{\text{causal}} \right) V.
Project back and add residual skip:
x1=x+AttnOut⋅WOx_1 = x + \text{AttnOut} \cdot W_O.
② Feed-Forward Sub-Layer (SwiGLU):
xnorm2=RMSNorm(x1)x_{\text{norm2}} = \text{RMSNorm}(x_1).
Compute gated activation with intermediate dimension dffn≈83dd_{\text{ffn}} \approx \frac{8}{3}d:
FFNOut=(SiLU(xnorm2Wgate)⊙(xnorm2Wup))Wdown\text{FFNOut} = \left( \text{SiLU}(x_{\text{norm2}} W_{\text{gate}}) \odot (x_{\text{norm2}} W_{\text{up}}) \right) W_{\text{down}}.
Add second residual skip:
xout=x1+FFNOutx_{\text{out}} = x_1 + \text{FFNOut}.

🏭 Production Trade-offs

Post-LN vs Pre-LN Historical Note: Original Vaswani Transformer placed LayerNorm after residual addition (xl+1=LN(x+F(x))x_{l+1} = \text{LN}(x + \mathcal{F}(x))). Modern models universally place normalization before sub-layers (Pre-LN / Pre-RMSNorm) to guarantee gradient stability.
⚠️ Common Interview Pitfalls
  • ✕
    Omitting the final LayerNorm/RMSNorm after the last Transformer block before the output unembedding head
  • ✕
    Using standard ReLU in modern Transformer MLPs instead of SwiGLU or GELU
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does the residual connection must bypass the normalization layer in Pre-LN architectures?
  • ?
    What architectural changes distinguish a Decoder block from an Encoder-Decoder cross-attention block?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-012: Architectural Paradigms: Encoder-Decoder vs Decoder-Only vs Encoder-Only📋 Back to BankNext →M4-014: Scaled Dot-Product Attention Formula and the Vital Role of the 1/√d_k Factor