M4-010M4: Sequences & TransformersSeq2Seq & Attention OriginsMedium
⚡Flashcard Drill➔Mastery:

Teacher Forcing and Exposure Bias in Autoregressive Sequence Generation

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-011 →
📐 Mathematical Definition
train: p(yt∣y<t∗,x);infer: p(yt∣y^<t,x);dist mismatch\text{train}:\ p(y_t|y^*_{<t},x);\qquad \text{infer}:\ p(y_t|\hat y_{<t},x);\qquad \text{dist mismatch}
⚡ One-Sentence Core Answer
Teacher forcing trains models by feeding ground-truth historical prefixes; exposure bias arises because inference feeds self-generated tokens, causing errors to compound rapidly.

📌 Key Takeaways

  • •
    Teacher Forcing: training step tt receives ground-truth target yt−1∗y_{t-1}^* as input; enables parallel training across all time steps
  • •
    Exposure Bias: during inference, the model receives its own potentially corrupted prediction y^t−1\hat{y}_{t-1}; out-of-distribution drift
  • •
    Compounding errors: a single erroneous token early in generation throws subsequent hidden states into unexplored regions

📐 Mathematical Derivations

Mathematical Formulations and Dynamics:
① Teacher Forcing Training Regime:
Target likelihood: max⁡θ∑t=1Tlog⁡P(yt∗∣y1:t−1∗,X;θ)\max_\theta \sum_{t=1}^T \log P(y_t^* \mid y_{1:t-1}^*, X; \theta).
- Advantage: Clean gradients, rapid convergence, and full temporal parallelization (in Transformers, causal masking computes all TT positions in a single forward pass).
- Flaw: The network is never trained on its own mistakes; it always assumes past history is 100% grammatically and semantically perfect.
② Inference Reality (Free Running):
Generated sequence: y^t∼P(⋅∣y^1:t−1,X;θ)\hat{y}_t \sim P(\cdot \mid \hat{y}_{1:t-1}, X; \theta).
- Distributional Shift: Let PdataP_{\text{data}} be the distribution of ground-truth prefixes, and PmodelP_{\text{model}} be the prefix distribution induced by sampling. In inference, inputs are drawn from Pmodel≠PdataP_{\text{model}} \ne P_{\text{data}}. If the model generates an awkward word at step 5, it enters an unfamiliar state not seen during training, triggering cascading hallucination.
③ Mitigation: Scheduled Sampling (Bengio et al., 2015):
At step tt, input token is selected via coin flip: xt={yt−1∗with probability ϵiy^t−1with probability 1−ϵix_t = \begin{cases} y_{t-1}^* & \text{with probability } \epsilon_i \\ \hat{y}_{t-1} & \text{with probability } 1 - \epsilon_i \end{cases}.
Decay schedule gradually anneals ϵi\epsilon_i from 1.01.0 (pure teacher forcing) to 00 (free running).

🏭 Production Trade-offs

RL / DPO Alternative: In modern LLM training, models undergo Supervised Fine-Tuning (SFT) with teacher forcing, followed by Reinforcement Learning from Human Feedback (RLHF / DPO) where models generate full unconstrained trajectories, directly eliminating exposure bias.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to use Scheduled Sampling in standard Transformer training; feeding self-generated tokens breaks causal mask parallelization and forces slow sequential unrolling
  • ✕
    Equating exposure bias with simple overfitting; exposure bias is a distribution shift between training conditioning and inference conditioning
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is Scheduled Sampling computationally incompatible with parallel Transformer training?
  • ?
    How does Direct Preference Optimization (DPO) alleviate exposure bias during LLM post-training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-009: Why Attention is Characterized as 'Differentiable Soft Alignment'📋 Back to BankNext →M4-011: Beam Search Decoding and Length Bias Normalization