M5-064M5: NLP & Large Language ModelsGRPO & Reasoning ModelsHard
⚡Flashcard Drill➔Mastery:
Process Reward Models (PRM) vs. Outcome Reward Models (ORM): Usage and Annotation Cost
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Outcome Reward Models (ORM) evaluate only final answer correctness with minimal annotation expense, whereas Process Reward Models (PRM) score every intermediate step to provide fine-grained credit assignment at the cost of massive human labeling overhead.
📌 Key Takeaways
- •ORM: assigns a single scalar score to the final token; simple and automated via unit tests/ground truth, but provides coarse credit assignment across long reasoning chains
- •PRM: assigns a step-level reward after every reasoning step; pinpoints exact error locations and powers Best-of- tree search, but requires expensive step-by-step annotation (PRM800K)
- •Industrial consensus: ORM dominates RL training (simpler, immune to step-level proxy hacking); PRM excels in test-time tree search (MCTS) to prune bad reasoning branches early
📐 Mathematical Derivations
Mathematical Mechanism: 1. Outcome Reward Model (ORM) Formulation: Given question and generated reasoning trajectory ending in answer : Evaluated once per sequence. If is wrong, the entire chain receives low reward, even if steps were brilliant. 2. Process Reward Model (PRM) Formulation: For each intermediate reasoning step , PRM evaluates the probability that step is mathematically sound and leads to a solvable state: The overall trajectory score is typically the product of step probabilities: A single fatal slip at step drives , causing trajectory reward to collapse immediately.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① The Annotation Cost Barrier: OpenAI's PRM800K required human math experts to inspect and label 800,000 individual reasoning steps, costing millions of dollars. Outcome verification (ORM / RLVR) runs automatically on execution sandboxes for fractions of a cent per problem. ② Automated PRM via Monte Carlo Rollouts (Math-Shepherd): Instead of human step labeling, automate PRM training via Monte Carlo rollouts: from intermediate step , sample completion paths to the end; the empirical fraction of completions that reach the correct answer defines the pseudo-label for step : . ③ Why PRM Struggles in RL Training: Using a neural PRM as the reward signal inside PPO/GRPO introduces more reward model exploitation surface area. Policies quickly learn subtle phrasing tricks that maximize step-level PRM scores without solving the problem. Consequently, frontier reasoning models (o1, R1) use ORM for RL training and reserve PRM for test-time search. ④ Pruning in Tree Search (MCTS): During inference, PRM allows beam search or Monte Carlo Tree Search to prune dead branches at step 3 instead of generating all 20 steps, saving test-time compute. ⑤ Interview Strategy: Contrast ORM vs PRM across definition, annotation cost, and search guidance, explain Monte Carlo rollouts for automated PRM labeling (Math-Shepherd), and justify why ORM dominates RL while PRM dominates search.
⚠️ Common Interview Pitfalls
- ✕Assuming PRM must be used during RL training (ORM with RLVR is significantly more robust and immune to step-level proxy hacking)
- ✕Relying on manual human annotation to scale PRM datasets (must use Monte Carlo rollout estimation like Math-Shepherd)
- ✕Ignoring that ORM cannot differentiate between genuine logical reasoning and a lucky guess
🎯 Interviewer Follow-up Inquiries
- ?How does Math-Shepherd use Monte Carlo completion rollouts to train a Process Reward Model without human step labels?
- ?Why does using a learned PRM inside an RL training loop increase the risk of reward hacking compared to binary ORM?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚