M5-034M5: NLP & Large Language ModelsAlignment & RLHFHard
⚡Flashcard Drill➔Mastery:
Reward Hacking and Mitigation Strategies in RLHF
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Reward hacking occurs when a policy exploits flaws in the proxy reward model to score high rewards without improving true response quality, mitigated through KL penalties, reward ensembles, length penalties, and iterative reward model updates.
📌 Key Takeaways
- •Goodhart's Law: 'When a measure becomes a target, it ceases to be a good measure'; reward models are imperfect proxy approximations of human preference
- •Typical manifestations: excessive verbosity (padding answers), sycophancy (flattering user misconceptions), repetitive stylistic formatting, and exploiting edge-case scoring bugs
- •Mitigation suite: KL divergence regularization against reference policy , Reward Model ensembling, explicit length penalty terms, and online iterative RM retraining
📐 Mathematical Derivations
Mathematical Mechanism: 1. The Optimization Discrepancy: Let be genuine human utility, and be the learned proxy reward model. As the policy is optimized aggressively against : The policy navigates into regions of token space where has high epistemic uncertainty (out-of-distribution), exploiting reward model over-generalization. 2. KL Regularization Constraint: Constrains policy to remain close to the pre-trained distribution : Where . If the policy drifts toward nonsensical high-reward patterns, the KL penalty explodes, halting the hack.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Verbosity Bias Mitigation: Because annotators favor long answers, policies learn that padding responses with bullet points increases reward. Mitigate by: (a) adding length penalty terms to reward: ; (b) training the reward model on length-normalized pairs or prompt-matched pairs of identical length. ② Reward Model Ensembling: Train independent reward models with different random seeds. Use the conservative lower bound as the optimization reward: Prevents the policy from exploiting an anomaly present in only one model. ③ Iterative Online RLHF: Periodically sample new policy completions, have humans/judges label preferences on current policy outputs, and retrain the reward model on this fresh on-policy distribution (preventing out-of-distribution drift). ④ Rule-Based Heuristic Filters: Combine neural reward models with deterministic rule checks (penalizing repetitive n-grams, profanity, or extreme length disparities). ⑤ Interview Strategy: Reference Goodhart's Law, detail the 4-part mitigation toolkit (KL penalty, length-penalized reward, RM ensembling, iterative online updates), and cite verbosity and sycophancy as canonical examples.
⚠️ Common Interview Pitfalls
- ✕Assuming rising reward model scores during RL training prove that response quality is improving
- ✕Setting the KL penalty coefficient too low, resulting in rapid linguistic degeneration
- ✕Treating reward hacking as a simple bug rather than a fundamental property of optimizing proxy objectives
🎯 Interviewer Follow-up Inquiries
- ?Why is verbosity bias the most pervasive form of reward hacking in commercial conversational LLMs?
- ?How does using the minimum across an ensemble of reward models prevent policy exploitation of reward peaks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚