M5-061M5: NLP & Large Language ModelsGRPO & Reasoning ModelsHard
⚡Flashcard Drill➔Mastery:
Cold Start Problem in Zero-RL / RLVR and Mitigation Strategies
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
The cold start problem occurs when a base model has a near-zero success rate on hard reasoning tasks, producing zero positive rewards and stalling RLVR exploration, solved through curated CoT seed demonstrations, prompt scaffolding, and difficulty curricula.
📌 Key Takeaways
- •Root mechanism: in binary verification (), if policy has pass rate on problem , all rollouts fail; advantages evaluate to zero, and policy gradients vanish
- •DeepSeek-R1 solution: cold-start SFT stage fine-tuning the base model on a few thousand high-quality long-CoT examples before initiating GRPO
- •Alternative mitigations: difficulty curriculum learning (starting with grade-school math), synthetic hints and sub-goal scaffolding, and rejection sampling initialization
📐 Mathematical Derivations
Mathematical Mechanism: 1. The Zero-Gradient Exploration Trap: In GRPO, the advantage for sample in group is: If problem is too difficult for current policy , all sampled candidates fail: . Consequently: The policy gradient evaluates to identically zero: The model receives zero directional feedback. Without an initial spark of success, RL exploration cannot bootstrap itself. 2. Cold-Start Initialization Bridge: Pre-train policy on a curated seed demonstration set : Raising initial pass rate from to ensures that across samples, the probability of sampling at least one correct path jumps from to , kickstarting the RL engine.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① DeepSeek-R1 Cold-Start Design: DeepSeek curated a few thousand high-quality reasoning examples with explicit readability constraints: (a) reasoning steps formatted inside
<think> ... </think>; (b) markdown headers; and (c) reflection checks. This eliminated the linguistic chaos and multi-language mixing observed in R1-Zero while providing the initial non-zero success rate needed for RLVR. ② Curriculum Learning Schedule: Start RLVR on high-success problems (GSM8K, simple LeetCode with ). As the model masters intermediate problem-solving patterns, gradually inject harder problems (MATH, Codeforces, AIME). The model transfers learned decomposition skills to maintain on difficult tiers. ③ Scaffolding with Partial Prompts: For intractable problems, provide the first 2-3 reasoning steps as prompt context; the model learns to complete the remaining steps, receiving reward. Gradually truncate the scaffolding over successive epochs until the model solves the problem from scratch. ④ Risk of Oversized Cold-Start Data: Training on too much SFT demonstration data ( examples) biases the model toward rigid memorization of human reasoning styles, restricting its ability to discover novel, super-human search paths during RL. Keep cold-start data small ( samples). ⑤ Interview Strategy: Formulate the zero-gradient mathematical trap when for all , explain how cold-start SFT raises pass rate past the threshold, and describe curriculum scheduling.⚠️ Common Interview Pitfalls
- ✕Attempting to launch RLVR directly on Olympiad math without cold-start seeding or curriculum (training stalls at step 0)
- ✕Using massive SFT datasets for cold start, which suppresses the model's ability to discover creative self-correction strategies
- ✕Failing to filter cold-start demonstrations for strict single-language and markdown formatting standards
🎯 Interviewer Follow-up Inquiries
- ?Why did DeepSeek-R1 use only a few thousand cold-start demonstrations rather than a traditional massive 500k-sample SFT corpus?
- ?How does prompt scaffolding mathematically bootstrap policy exploration on zero-pass-rate problems?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚