M5-058M5: NLP & Large Language ModelsGRPO & Reasoning ModelsHard
⚡Flashcard Drill➔Mastery:

Reasoning Model Training Paradigm: Long CoT and Reinforcement Learning

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-059 →
📐 Mathematical Definition
reasoning model: long CoT emerges from RLVR;length↑ with accuracy↑ (up to a point)\text{reasoning model}:\ \text{long CoT}\ \text{emerges from RLVR};\qquad \text{length}\uparrow\ \text{with}\ \text{accuracy}\uparrow\ \text{(up to a point)}
⚡ One-Sentence Core Answer
Modern reasoning models are trained by pairing high-capacity base models with reinforcement learning over verifiable rewards, autonomously incentivizing extended Chain-of-Thought generation, self-correction, and systematic hypothesis testing.

📌 Key Takeaways

  • •
    Paradigm shift: moving beyond static SFT imitation toward self-directed reinforcement learning exploration (OpenAI o1, DeepSeek-R1)
  • •
    The RLVR engine: training with large group sampling (GRPO) rewarded strictly on execution correctness (math ground truth, code unit tests)
  • •
    Emergent cognitive behaviors: without human prompts, models spontaneously learn to generate internal thinking tokens (<think>), double-check equations, backtrack from dead ends, and decompose hard problems

📐 Mathematical Derivations

Mathematical Mechanism: 1. The Extended Reasoning Formulation: The model factorizes the conditional probability into an internal reasoning trace R=[r1,…,rK]R = [r_1, \dots, r_K] followed by final answer AA: P(A∣X)=∑RP(R∣X)P(A∣X,R)P(A \mid X) = \sum_{R} P(R \mid X) P(A \mid X, R) In standard models, K≈0K \approx 0 (direct prediction). In reasoning models, RL incentives cause length KK to expand dynamically (K∈[1000,20000]K \in [1000, 20000] tokens). 2. RL Exploration & The 'Aha Moment': When trained via GRPO on difficult Olympiad math: - Step 1: Early in training, the model attempts quick answers and receives reward 0.00.0. - Step 2: By random exploration, a rollout includes a reflective verification phrase ('Wait, let me recalculate the determinant of matrix MM'). - Step 3: The recalculation catches an algebraic slip, leading to a correct answer and reward 1.01.0. - Step 4: Group advantage strongly reinforces the self-reflection tokens. Over thousands of gradient steps, the policy systematically internalizes reflection and backtracking circuits.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① The Cold-Start Prerequisite (DeepSeek-R1 vs R1-Zero): - R1-Zero (pure RL from base model) exhibited messy formatting, multi-language mixing (switching between Chinese and English mid-sentence), and unreadable verbosity. - DeepSeek-R1 solved this by seeding the base model with a small, curated cold-start dataset (∼1,000\sim 1,000 multi-turn long-CoT human demonstrations) before launching RL, enforcing clean markdown formatting and single-language coherence from step 1. ② Outcome vs Process Rewards: While Process Reward Models (PRMs) provide step-level supervision, they suffer from annotation cost and proxy reward hacking. Modern breakthrough reasoning models rely primarily on Outcome Rewards (RLVR), proving that outcome verification is sufficient to induce process correctness. ③ Inference-Time Latency Trade-off: Reasoning models trade inference FLOPs for accuracy: generating 10,000 reasoning tokens takes 20-60 seconds and multiplies API costs by 10×10\times to 50×50\times. Serving architectures require adaptive thinking budgets. ④ Generalization Beyond Math & Code: Once reasoning circuits are internalized, models exhibit dramatic improvements on general STEM, law, and logical planning benchmarks. ⑤ Interview Strategy: Detail the 3-step paradigm (Cold-start SFT →\to Large-scale RLVR with GRPO →\to Rejection-sampling distillation), explain the 'Aha moment' self-correction mechanism, and contrast R1-Zero with DeepSeek-R1.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to hand-craft thousands of complex reasoning rules instead of letting RL discover optimal search strategies
  • ✕
    Omitting cold-start SFT initialization, leading to language-mixing and chaotic output formatting (the R1-Zero failure mode)
  • ✕
    Using neural reward models for reasoning tasks where programmatic unit tests and math solvers are available
🎯 Interviewer Follow-up Inquiries
  • ?
    What causes pure RLVR models (like DeepSeek-R1-Zero) to spontaneously mix languages during extended reasoning?
  • ?
    How does seeding a base model with 1,000 cold-start CoT examples prevent formatting chaos during subsequent RL?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-057: DAPO (Decoupled Advantage Policy Optimization) Improvement Highlights📋 Back to BankNext →M5-059: Evaluation and Cost Trade-offs of Reasoning Models