M4-012M4: Sequences & TransformersSeq2Seq & Attention OriginsHard
⚡Flashcard Drill➔Mastery:
Architectural Paradigms: Encoder-Decoder vs Decoder-Only vs Encoder-Only
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Encoder-Only builds bidirectional representations (BERT); Encoder-Decoder performs sequence mapping (T5); Decoder-Only unifies comprehension and generation (GPT/LLaMA), dominating modern LLMs.
📌 Key Takeaways
- •Encoder-Only (BERT/RoBERTa): bidirectional attention, no causal mask; optimal for classification, NER, and dense semantic embeddings
- •Encoder-Decoder (T5/BART): bidirectional encoder + causal cross-attention decoder; specialized for translation and summarization
- •Decoder-Only (GPT/LLaMA): causal autoregressive masking throughout; unified architecture dominating large language models
📐 Mathematical Derivations
Structural Comparison Matrix:
| Feature | Encoder-Only (BERT) | Encoder-Decoder (T5) | Decoder-Only (LLaMA/GPT) |
| :--- | :--- | :--- | :--- |
| Attention Mask | Full Bidirectional () | Enc: Bidirectional; Dec: Causal | Strictly Causal Lower-Triangular |
| Pre-training Objective | Masked LM (MLM) | Span Corruption / Denoising | Next-Token Prediction (Causal LM) |
| Generation Capability | Non-autoregressive (poor) | Autoregressive (separate cross-attn) | Autoregressive (native) |
| Computational Reuse | Zero KV-cache | KV-cache on decoder only | KV-cache across all layers |
Why Decoder-Only Dominates Modern LLMs:
1. Unified Next-Token Objective: In-context learning and multi-task prompt instructions require zero architectural task heads.
2. Hardware & KV Cache Simplicity: Single homogenous block design scales cleanly across 3D parallelism; eliminates cross-attention state management.
3. Empirical Scaling Laws: When compute budget is held fixed, Decoder-Only models achieve equal or superior downstream zero-shot reasoning compared to Encoder-Decoder architectures.
| Feature | Encoder-Only (BERT) | Encoder-Decoder (T5) | Decoder-Only (LLaMA/GPT) |
| :--- | :--- | :--- | :--- |
| Attention Mask | Full Bidirectional () | Enc: Bidirectional; Dec: Causal | Strictly Causal Lower-Triangular |
| Pre-training Objective | Masked LM (MLM) | Span Corruption / Denoising | Next-Token Prediction (Causal LM) |
| Generation Capability | Non-autoregressive (poor) | Autoregressive (separate cross-attn) | Autoregressive (native) |
| Computational Reuse | Zero KV-cache | KV-cache on decoder only | KV-cache across all layers |
Why Decoder-Only Dominates Modern LLMs:
1. Unified Next-Token Objective: In-context learning and multi-task prompt instructions require zero architectural task heads.
2. Hardware & KV Cache Simplicity: Single homogenous block design scales cleanly across 3D parallelism; eliminates cross-attention state management.
3. Empirical Scaling Laws: When compute budget is held fixed, Decoder-Only models achieve equal or superior downstream zero-shot reasoning compared to Encoder-Decoder architectures.
🏭 Production Trade-offs
When to choose Encoder-Only: When building production embedding search engines (Sentence-BERT, dense retrievers) or text classification APIs with strict microsecond latency budgets, Encoder-Only models are more efficient than LLMs.
⚠️ Common Interview Pitfalls
- ✕Using a Decoder-Only LLM for simple token classification or sentence embedding when an Encoder-Only model is faster and cheaper
- ✕Assuming Encoder-Decoder models are obsolete; they remain competitive for specialized machine translation (e.g., NLLB)
🎯 Interviewer Follow-up Inquiries
- ?Why does the Next-Token Prediction objective in Decoder-Only models naturally generalize to multi-task in-context learning?
- ?What specific memory advantages does Decoder-Only KV-caching offer over Encoder-Decoder architectures during inference?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚