M5-064M5: NLP & Large Language ModelsGRPO & Reasoning ModelsHard
⚡Flashcard Drill➔Mastery:

Process Reward Models (PRM) vs. Outcome Reward Models (ORM): Usage and Annotation Cost

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-065 →
📐 Mathematical Definition
PRM: r(s1,…,sk) per step;cost: step-level labels are expensive\text{PRM}:\ r(s_1,\dots,s_k)\ \text{per step};\qquad \text{cost}:\ \text{step-level labels are expensive}
⚡ One-Sentence Core Answer
Outcome Reward Models (ORM) evaluate only final answer correctness with minimal annotation expense, whereas Process Reward Models (PRM) score every intermediate step to provide fine-grained credit assignment at the cost of massive human labeling overhead.

📌 Key Takeaways

  • •
    ORM: assigns a single scalar score R(x,y)R(x, y) to the final token; simple and automated via unit tests/ground truth, but provides coarse credit assignment across long reasoning chains
  • •
    PRM: assigns a step-level reward r(st)r(s_t) after every reasoning step; pinpoints exact error locations and powers Best-of-NN tree search, but requires expensive step-by-step annotation (PRM800K)
  • •
    Industrial consensus: ORM dominates RL training (simpler, immune to step-level proxy hacking); PRM excels in test-time tree search (MCTS) to prune bad reasoning branches early

📐 Mathematical Derivations

Mathematical Mechanism: 1. Outcome Reward Model (ORM) Formulation: Given question xx and generated reasoning trajectory y=[s1,s2,…,sK]y = [s_1, s_2, \dots, s_K] ending in answer AA: RORM(x,y)=P(Answer is correct∣x,y)∈[0,1]R_{\text{ORM}}(x, y) = P(\text{Answer is correct} \mid x, y) \in [0, 1] Evaluated once per sequence. If AA is wrong, the entire chain receives low reward, even if steps s1,…,sK−1s_1, \dots, s_{K-1} were brilliant. 2. Process Reward Model (PRM) Formulation: For each intermediate reasoning step k∈{1,…,K}k \in \{1, \dots, K\}, PRM evaluates the probability that step sks_k is mathematically sound and leads to a solvable state: rPRM(sk)=P(sk is correct and productive∣x,s<k)∈[0,1]r_{\text{PRM}}(s_k) = P(s_k \text{ is correct and productive} \mid x, s_{<k}) \in [0, 1] The overall trajectory score is typically the product of step probabilities: R(y)=∏k=1KrPRM(sk)R(y) = \prod_{k=1}^K r_{\text{PRM}}(s_k) A single fatal slip at step jj drives r(sj)→0r(s_j) \to 0, causing trajectory reward to collapse immediately.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① The Annotation Cost Barrier: OpenAI's PRM800K required human math experts to inspect and label 800,000 individual reasoning steps, costing millions of dollars. Outcome verification (ORM / RLVR) runs automatically on execution sandboxes for fractions of a cent per problem. ② Automated PRM via Monte Carlo Rollouts (Math-Shepherd): Instead of human step labeling, automate PRM training via Monte Carlo rollouts: from intermediate step sks_k, sample MM completion paths to the end; the empirical fraction of completions that reach the correct answer defines the pseudo-label for step sks_k: r(sk)≈1M∑m=1MI(correct)r(s_k) \approx \frac{1}{M} \sum_{m=1}^M \mathbb{I}(\text{correct}). ③ Why PRM Struggles in RL Training: Using a neural PRM as the reward signal inside PPO/GRPO introduces K×K\times more reward model exploitation surface area. Policies quickly learn subtle phrasing tricks that maximize step-level PRM scores without solving the problem. Consequently, frontier reasoning models (o1, R1) use ORM for RL training and reserve PRM for test-time search. ④ Pruning in Tree Search (MCTS): During inference, PRM allows beam search or Monte Carlo Tree Search to prune dead branches at step 3 instead of generating all 20 steps, saving test-time compute. ⑤ Interview Strategy: Contrast ORM vs PRM across definition, annotation cost, and search guidance, explain Monte Carlo rollouts for automated PRM labeling (Math-Shepherd), and justify why ORM dominates RL while PRM dominates search.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming PRM must be used during RL training (ORM with RLVR is significantly more robust and immune to step-level proxy hacking)
  • ✕
    Relying on manual human annotation to scale PRM datasets (must use Monte Carlo rollout estimation like Math-Shepherd)
  • ✕
    Ignoring that ORM cannot differentiate between genuine logical reasoning and a lucky guess
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Math-Shepherd use Monte Carlo completion rollouts to train a Process Reward Model without human step labels?
  • ?
    Why does using a learned PRM inside an RL training loop increase the risk of reward hacking compared to binary ORM?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-063: Reasoning Model Distillation: Transferring Long CoT into Compact Models📋 Back to BankNext →M5-065: Efficiency Challenges in Reasoning Models: Overthinking and Length Adaptation