M5-061M5: NLP & Large Language ModelsGRPO & Reasoning ModelsHard
⚡Flashcard Drill➔Mastery:

Cold Start Problem in Zero-RL / RLVR and Mitigation Strategies

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-062 →
📐 Mathematical Definition
cold start: base model→RL ⇒ format collapse, language mixing;fix: SFT warmup\text{cold start}:\ \text{base model}\to\text{RL}\ \Rightarrow\ \text{format collapse},\ \text{language mixing};\qquad \text{fix}:\ \text{SFT warmup}
⚡ One-Sentence Core Answer
The cold start problem occurs when a base model has a near-zero success rate on hard reasoning tasks, producing zero positive rewards and stalling RLVR exploration, solved through curated CoT seed demonstrations, prompt scaffolding, and difficulty curricula.

📌 Key Takeaways

  • •
    Root mechanism: in binary verification (R∈{0,1}R \in \{0, 1\}), if policy πθ\pi_\theta has pass rate p=0p=0 on problem xx, all GG rollouts fail; advantages evaluate to zero, and policy gradients vanish
  • •
    DeepSeek-R1 solution: cold-start SFT stage fine-tuning the base model on a few thousand high-quality long-CoT examples before initiating GRPO
  • •
    Alternative mitigations: difficulty curriculum learning (starting with grade-school math), synthetic hints and sub-goal scaffolding, and rejection sampling initialization

📐 Mathematical Derivations

Mathematical Mechanism: 1. The Zero-Gradient Exploration Trap: In GRPO, the advantage for sample ii in group GG is: A^i=Ri−RˉσR+ϵ\hat{A}_i = \frac{R_i - \bar{R}}{\sigma_R + \epsilon} If problem xx is too difficult for current policy πθ\pi_\theta, all sampled candidates fail: R1=R2=⋯=RG=0R_1 = R_2 = \dots = R_G = 0. Consequently: Rˉ=0,σR=0  ⟹  A^i=0−00+ϵ=0∀i∈{1,…,G}\bar{R} = 0, \quad \sigma_R = 0 \implies \hat{A}_i = \frac{0 - 0}{0 + \epsilon} = 0 \quad \forall i \in \{1, \dots, G\} The policy gradient evaluates to identically zero: ∇θLGRPO=0\nabla_\theta \mathcal{L}_{\text{GRPO}} = \mathbf{0} The model receives zero directional feedback. Without an initial spark of success, RL exploration cannot bootstrap itself. 2. Cold-Start Initialization Bridge: Pre-train policy θ\theta on a curated seed demonstration set Dcold_start\mathcal{D}_{\text{cold\_start}}: θinit=arg⁡max⁡θ∑(x,R,y)∈Dcold_startlog⁡Pθ(R,y∣x)\theta_{\text{init}} = \arg\max_\theta \sum_{(x, R, y) \in \mathcal{D}_{\text{cold\_start}}} \log P_\theta(R, y \mid x) Raising initial pass rate p(x)p(x) from 0.000.00 to ≥0.05\ge 0.05 ensures that across G=16G=16 samples, the probability of sampling at least one correct path jumps from 0%0\% to 1−(0.95)16≈56%1 - (0.95)^{16} \approx 56\%, kickstarting the RL engine.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① DeepSeek-R1 Cold-Start Design: DeepSeek curated a few thousand high-quality reasoning examples with explicit readability constraints: (a) reasoning steps formatted inside <think> ... </think>; (b) markdown headers; and (c) reflection checks. This eliminated the linguistic chaos and multi-language mixing observed in R1-Zero while providing the initial non-zero success rate needed for RLVR. ② Curriculum Learning Schedule: Start RLVR on high-success problems (GSM8K, simple LeetCode with p>0.3p > 0.3). As the model masters intermediate problem-solving patterns, gradually inject harder problems (MATH, Codeforces, AIME). The model transfers learned decomposition skills to maintain p>0p > 0 on difficult tiers. ③ Scaffolding with Partial Prompts: For intractable problems, provide the first 2-3 reasoning steps as prompt context; the model learns to complete the remaining steps, receiving reward. Gradually truncate the scaffolding over successive epochs until the model solves the problem from scratch. ④ Risk of Oversized Cold-Start Data: Training on too much SFT demonstration data (>50k>50\text{k} examples) biases the model toward rigid memorization of human reasoning styles, restricting its ability to discover novel, super-human search paths during RL. Keep cold-start data small (∼1k–5k\sim 1\text{k}\text{--}5\text{k} samples). ⑤ Interview Strategy: Formulate the zero-gradient mathematical trap when Ri=0R_i = 0 for all ii, explain how cold-start SFT raises pass rate past the 1−(1−p)G1 - (1-p)^G threshold, and describe curriculum scheduling.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to launch RLVR directly on Olympiad math without cold-start seeding or curriculum (training stalls at step 0)
  • ✕
    Using massive SFT datasets for cold start, which suppresses the model's ability to discover creative self-correction strategies
  • ✕
    Failing to filter cold-start demonstrations for strict single-language and markdown formatting standards
🎯 Interviewer Follow-up Inquiries
  • ?
    Why did DeepSeek-R1 use only a few thousand cold-start demonstrations rather than a traditional massive 500k-sample SFT corpus?
  • ?
    How does prompt scaffolding mathematically bootstrap policy exploration on zero-pass-rate problems?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-060: Group Size and Sampling Efficiency in GRPO📋 Back to BankNext →M5-062: Length Control and Length Penalties in Long CoT Reasoning