M4-012M4: Sequences & TransformersSeq2Seq & Attention OriginsHard
⚡Flashcard Drill➔Mastery:
Architectural Paradigms: Encoder-Decoder vs Decoder-Only vs Encoder-Only
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Encoder-Only builds bidirectional representations (BERT); Encoder-Decoder performs sequence mapping (T5); Decoder-Only unifies comprehension and generation (GPT/LLaMA), dominating modern LLMs.
📌 Key Takeaways
- •Encoder-Only (BERT/RoBERTa): bidirectional attention, no causal mask; optimal for classification, NER, and dense semantic embeddings
- •Encoder-Decoder (T5/BART): bidirectional encoder + causal cross-attention decoder; specialized for translation and summarization
- •Decoder-Only (GPT/LLaMA): causal autoregressive masking throughout; unified architecture dominating large language models
📐 Mathematical Derivations
Structural Comparison Matrix:
| Feature | Encoder-Only (BERT) | Encoder-Decoder (T5) | Decoder-Only (LLaMA/GPT) |
| :--- | :--- | :--- | :--- |
| Attention Mask | Full Bidirectional () | Enc: Bidirectional; Dec: Causal | Strictly Causal Lower-Triangular |
| Pre-training Objective | Masked LM (MLM) | Span Corruption / Denoising | Next-Token Prediction (Causal LM) |
| Generation Capability | Non-autoregressive (poor) | Autoregressive (separate cross-attn) | Autoregressive (native) |
| Computational Reuse | Zero KV-cache | KV-cache on decoder only | KV-cache across all layers |
Why Decoder-Only Dominates Modern LLMs:
1. Unified Next-Token Objective: In-context learning and multi-task prompt instructions require zero architectural task heads.
2. Hardware & KV Cache Simplicity: Single homogenous block design scales cleanly across 3D parallelism; eliminates cross-attention state management.
3. Empirical Scaling Laws: When compute budget is held fixed, Decoder-Only models achieve equal or superior downstream zero-shot reasoning compared to Encoder-Decoder architectures.
| Feature | Encoder-Only (BERT) | Encoder-Decoder (T5) | Decoder-Only (LLaMA/GPT) |
| :--- | :--- | :--- | :--- |
| Attention Mask | Full Bidirectional () | Enc: Bidirectional; Dec: Causal | Strictly Causal Lower-Triangular |
| Pre-training Objective | Masked LM (MLM) | Span Corruption / Denoising | Next-Token Prediction (Causal LM) |
| Generation Capability | Non-autoregressive (poor) | Autoregressive (separate cross-attn) | Autoregressive (native) |
| Computational Reuse | Zero KV-cache | KV-cache on decoder only | KV-cache across all layers |
Why Decoder-Only Dominates Modern LLMs:
1. Unified Next-Token Objective: In-context learning and multi-task prompt instructions require zero architectural task heads.
2. Hardware & KV Cache Simplicity: Single homogenous block design scales cleanly across 3D parallelism; eliminates cross-attention state management.
3. Empirical Scaling Laws: When compute budget is held fixed, Decoder-Only models achieve equal or superior downstream zero-shot reasoning compared to Encoder-Decoder architectures.
🏭 Production Trade-offs
When to choose Encoder-Only: When building production embedding search engines (Sentence-BERT, dense retrievers) or text classification APIs with strict microsecond latency budgets, Encoder-Only models are more efficient than LLMs.
⚠️ Common Interview Pitfalls
- ✕Using a Decoder-Only LLM for simple token classification or sentence embedding when an Encoder-Only model is faster and cheaper
- ✕Assuming Encoder-Decoder models are obsolete; they remain competitive for specialized machine translation (e.g., NLLB)
🎯 Interviewer Follow-up Inquiries
- ?Why does the Next-Token Prediction objective in Decoder-Only models naturally generalize to multi-task in-context learning?
- ?What specific memory advantages does Decoder-Only KV-caching offer over Encoder-Decoder architectures during inference?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚
Associated Knowledge Base Guides & Mindmaps
Explore the comprehensive technical article, exam cards, and global architecture tree.
中文速览与面试要点
比较编码器-解码器、仅解码器与仅编码器架构的适用场景
编码器-解码器适合序列到序列转换;仅编码器适合理解/表示;仅解码器适合生成与通用任务,已成为 LLM 主流。
核心要点
- 数学机理:三者的差异在于 注意力 mask 与信息流方向 。 仅编码器(Encoder-only) ——全向(非因果)注意力,每个位置可看所有位置,输出'上下文相关的表示';适合 理解类 任务(分类、NER、抽取式 QA、句子嵌入/检索),但 不能直接生成 变长序列(除非加任务头)。代表:BERT、RoBERTa、DeBERTa、以及嵌入模型(E5、BGE)。 仅解码器(Decoder-only) ——因果(下三角)注意力,每个位置只能看左侧;训练目标就是 next-token 预测。 优势 :(a) 训练信号密集(每个 token 都是一个预测任务)、(b) 可 统一所有任务为'条件续写' (分类=续写标签、翻译=续写译文、抽取=续写答案)、(c) 无需为每个任务设计专门的头部架构、(d) in-context learning 与 few-shot 能力天然涌现。代表:GPT 系、LLaMA、Qwen。 编码器-解码器(Encoder-Decoder) ——双向编码器 + 因果解码器,中间用 交叉注意力 让解码器查询编码器表示;适合 明确的序列到序列转换 (翻译、摘要、语音识别),优点是编码器可双向建模源序列(对源的理解更充分)、解码器与源长度解耦(不必把源也自回归地读一遍)。代表:T5、BART、mT5、Whisper。 选择逻辑 ——若任务是'给定输入产生不同长度的输出'且输入需要深度双向理解(翻译/摘要/ASR),Enc-Dec 有结构优势;若任务是通用、需一个模型覆盖多任务,Dec-only 更灵活;若只需表示/分类,Enc-only 最经济。
- 深度剖析与工程权衡:① Dec-only 胜出的现实原因 ——(a) 规模效应:同一套架构可无限堆叠、同一目标可吃所有文本数据,避免了'为每个任务设计架构'的工程成本;(b) in-context learning 使'任务'可由 prompt 指定,无需改架构;(c) 工程统一:训练、推理、优化只需维护一条链路。② Enc-Dec 的结构优势仍在 ——(a) 交叉注意力 让解码器每步访问完整源表示,对'源需要精细对齐'的任务(翻译)更有效;(b) 编码器可双向,对源的理解强于'因果读取源'的 Dec-only;(c) 输入与输出长度解耦,长输入 + 短输出的任务(摘要、检索增强生成)计算更省。③ Dec-only 的'伪编码器' ——通过把输入放在前缀、让注意力对它双向可见(prefix-LM 风格)或用专门的注意力 mask,Dec-only 也可获得部分双向理解能力;这是'架构趋同'的体现。④ 混合架构 ——如 UL2/Flan-T5 用多种去噪目标统一 Enc-Dec;而'解码器 + 检索编码器'(RAG)是 Enc-only 与 Dec-only 的组合。⑤ 多模态的启示 ——VLM 常用'ViT 编码器 + LLM 解码器'(Enc-Dec 结构),因为图像需要双向的全局理解、文本需要因果生成;这印证了'理解用编码器、生成用解码器'的分工。⑥ 面试要点 ——被问'三种架构怎么选',应按' 任务类型(转换/理解/通用)+ 注意力方向(因果/双向)+ 交叉注意力的必要性 '三层回答;并主动说明'Dec-only 的胜出主要是工程与规模原因,而非结构上全面更优'。
- Enc-Dec:双向编码 + 因果解码 + 交叉注意力(T5/BART)
- Enc-only:双向注意力,适合分类/标注/检索(BERT)
- Dec-only:因果注意力,可统一所有任务为'续写'(GPT/LLaMA)
常见陷阱
- 认为 Dec-only 结构上全面优于 Enc-Dec
- 在需要双向理解源序列的任务上忽略 Enc-Dec 的结构优势