M4-012M4: Sequences & TransformersSeq2Seq & Attention OriginsHard
⚡Flashcard Drill➔Mastery:

Architectural Paradigms: Encoder-Decoder vs Decoder-Only vs Encoder-Only

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-013 →
📐 Mathematical Definition
Enc-Dec: p(y∣x)=fdec(fenc(x));Dec-only: p(x)=∏tp(xt∣x<t);Enc-only: p(y∣x)=g(f(x))\text{Enc-Dec}:\ p(y|x)=f_{\text{dec}}(f_{\text{enc}}(x));\quad \text{Dec-only}:\ p(x)=\prod_t p(x_t|x_{<t});\quad \text{Enc-only}:\ p(y|x)=g(f(x))
⚡ One-Sentence Core Answer
Encoder-Only builds bidirectional representations (BERT); Encoder-Decoder performs sequence mapping (T5); Decoder-Only unifies comprehension and generation (GPT/LLaMA), dominating modern LLMs.

📌 Key Takeaways

  • •
    Encoder-Only (BERT/RoBERTa): bidirectional attention, no causal mask; optimal for classification, NER, and dense semantic embeddings
  • •
    Encoder-Decoder (T5/BART): bidirectional encoder + causal cross-attention decoder; specialized for translation and summarization
  • •
    Decoder-Only (GPT/LLaMA): causal autoregressive masking throughout; unified architecture dominating large language models

📐 Mathematical Derivations

Structural Comparison Matrix:
| Feature | Encoder-Only (BERT) | Encoder-Decoder (T5) | Decoder-Only (LLaMA/GPT) |
| :--- | :--- | :--- | :--- |
| Attention Mask | Full Bidirectional (N×NN \times N) | Enc: Bidirectional; Dec: Causal | Strictly Causal Lower-Triangular |
| Pre-training Objective | Masked LM (MLM) | Span Corruption / Denoising | Next-Token Prediction (Causal LM) |
| Generation Capability | Non-autoregressive (poor) | Autoregressive (separate cross-attn) | Autoregressive (native) |
| Computational Reuse | Zero KV-cache | KV-cache on decoder only | KV-cache across all layers |
Why Decoder-Only Dominates Modern LLMs:
1. Unified Next-Token Objective: In-context learning and multi-task prompt instructions require zero architectural task heads.
2. Hardware & KV Cache Simplicity: Single homogenous block design scales cleanly across 3D parallelism; eliminates cross-attention state management.
3. Empirical Scaling Laws: When compute budget is held fixed, Decoder-Only models achieve equal or superior downstream zero-shot reasoning compared to Encoder-Decoder architectures.

🏭 Production Trade-offs

When to choose Encoder-Only: When building production embedding search engines (Sentence-BERT, dense retrievers) or text classification APIs with strict microsecond latency budgets, Encoder-Only models are 10×10\times more efficient than LLMs.
⚠️ Common Interview Pitfalls
  • ✕
    Using a Decoder-Only LLM for simple token classification or sentence embedding when an Encoder-Only model is faster and cheaper
  • ✕
    Assuming Encoder-Decoder models are obsolete; they remain competitive for specialized machine translation (e.g., NLLB)
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does the Next-Token Prediction objective in Decoder-Only models naturally generalize to multi-task in-context learning?
  • ?
    What specific memory advantages does Decoder-Only KV-caching offer over Encoder-Decoder architectures during inference?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-011: Beam Search Decoding and Length Bias Normalization📋 Back to BankNext →M4-013: Complete Structural Anatomy of a Modern Transformer Block