M5-038M5: NLP & Large Language ModelsAlignment & RLHFHard
⚡Flashcard Drill➔Mastery:
On-Policy Sampling vs. Off-Policy Data in RLHF
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
RLHF requires on-policy sampling so that policy updates correspond strictly to the current distribution of generated tokens, while off-policy buffer reuse introduces severe distribution shifts and advantage estimation bias.
📌 Key Takeaways
- •On-policy requirement: policy gradient theorems require expectations taken over trajectories generated by the current policy:
- •Off-policy drift: if rollouts are collected from an older policy and reused for multiple PPO epochs, importance sampling weights explode or vanish, destabilizing updates
- •Engineering tension: on-policy sampling requires continuous, expensive generation rollouts during training, making RLHF to slower than offline supervised learning
📐 Mathematical Derivations
Mathematical Mechanism: 1. Importance Sampling Correction: When evaluating policy using trajectories sampled from behavior policy (off-policy): In language models where , the cumulative importance weight is a product of hundreds of ratios: If drifts slightly from , suffers from exponential variance explosion: either collapsing to zero for almost all samples or exploding to infinity on a single outlier sample, destroying gradient updates. 2. PPO Off-Policy Epoch Horizon: PPO allows mild off-policy reuse by collecting a batch with , training for 1-4 gradient epochs with clipping (), and immediately discarding the batch to collect fresh on-policy samples.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Rollout Freshness vs GPU Throughput: To maximize throughput, engineers want to train on a generated batch for many epochs. In LLMs, training for epochs on the same rollout batch leads to immediate policy collapse due to importance weight divergence. Batches must be refreshed constantly. ② Decoupled Asynchronous Rollouts: Frameworks like Ray-RLHF run dedicated inference engines (vLLM) that continuously generate rollouts into a shared queue while trainer nodes pull the freshest rollouts, overlapping generation and training without falling behind on-policy freshness. ③ Contrast with DPO (Offline Off-Policy): DPO is fully offline: it trains on static preference datasets without any on-policy generation rollouts. However, offline DPO suffers from distribution shift because the student never learns to correct its own generation errors (addressed by Online DPO). ④ Experience Replay Buffers in RLHF: Traditional RL experience replay buffers (storing samples across thousands of past steps) fail completely in LLM RLHF due to the exponential length product . Replay buffers are strictly restricted to 1 step. ⑤ Interview Strategy: Write down the cumulative sequence importance sampling formula , explain why variance explodes exponentially with sequence length , and justify why PPO restricts batch reuse to 1-2 epochs.
⚠️ Common Interview Pitfalls
- ✕Attempting to reuse rollout data across dozens of PPO training epochs to save compute (causes immediate divergence)
- ✕Using a traditional deep Q-learning experience replay buffer for full autoregressive LLM trajectories
- ✕Assuming DPO is on-policy (standard DPO is completely offline and off-policy)
🎯 Interviewer Follow-up Inquiries
- ?Why does the product of token-level importance weights make multi-epoch off-policy training practically impossible in LLMs?
- ?How does Online / Iterative DPO bridge the gap between offline simplicity and on-policy freshness?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚