M5-053M5: NLP & Large Language ModelsDPO Family (DPO / KTO / ORPO)Medium
⚡Flashcard Drill➔Mastery:
SFT and DPO Hybrid Training Strategies
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Co-training DPO with an auxiliary SFT loss over winning completions prevents likelihood displacement and language degeneration by anchoring the policy to high generative probability on positive demonstrations.
📌 Key Takeaways
- •The degeneration problem: pure DPO loss optimizes relative margins and often decreases the absolute probability of both completions, causing perplexity to explode and language fluency to degrade
- •Co-training objective: adds a weighted SFT loss term on winning responses:
- •Empirical stability: stabilizes training across multiple epochs, maintains low validation perplexity, and preserves pre-trained knowledge representations
📐 Mathematical Derivations
Mathematical Mechanism: 1. Likelihood Displacement in Pure DPO: DPO gradient with respect to policy : If drops by 2 nats while drops by 6 nats, DPO loss decreases by 4 nats. The policy satisfies the relative preference margin while simultaneously becoming less likely to generate the correct answer in greedy decoding! 2. Hybrid SFT + DPO Formulation: Adding supervised cross-entropy over : The gradient of the hybrid loss becomes: Because the coefficient on is strictly larger than on by margin , the optimizer is mathematically forced to increase the absolute generation probability of the winning response.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Hyperparameter Tuning $\gamma$: Setting provides optimal regularization. If is too small (), likelihood displacement still occurs; if is too large (), SFT dominates and preference separation is muted. ② Pre-training Data Replay Integration: In addition to , adding pre-training general text replay () completely protects against catastrophic forgetting during extended alignment. ③ Two-Phase vs Single-Phase: Standard practice runs sequential SFT followed by DPO. Hybrid co-training allows teams to run SFT and DPO simultaneously in a single unified training stage (as in ORPO), cutting post-training compute time in half. ④ Validation Metric Monitoring: During hybrid training, track both preference classification accuracy and validation cross-entropy loss on clean held-out text; a healthy run exhibits rising accuracy and steady or decreasing perplexity. ⑤ Interview Strategy: Explain why pure DPO suffers from likelihood displacement, write out the hybrid gradient showing how forces positive updates on , and cite empirical stability gains.
⚠️ Common Interview Pitfalls
- ✕Training pure DPO for multiple epochs without an SFT regularizer, leading to sudden language collapse
- ✕Setting so large that preference alignment is drowned out by supervised imitation
- ✕Computing the SFT regularizer on both winning and losing responses (must apply strictly to )
🎯 Interviewer Follow-up Inquiries
- ?How does the auxiliary SFT coefficient prevent the model from drifting into repetitive degeneration loops?
- ?What is the mathematical relationship between hybrid SFT+DPO and Odds Ratio Preference Optimization (ORPO)?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚