Process Reward Models PRM & Monte Carlo

Process Supervision Reward Models (PRMs) & Step-Level Monte Carlo Credit Assignment revolutionize mathematical reasoning by replacing sparse final trajectory scores with dense step-by-step verification; while Outcome Reward Models (ORMs) score only the final answer (failing to penalize hallucinated reasoning steps that stumble onto correct answers by chance), PRMs evaluate the rigorous correctness $r(s_t) \in [0, 1]$ of every individual deduction step; automated weak supervision (Math-Shepherd framework) annotates step-level ground-truth by launching $K$ independent Monte Carlo rollouts from intermediate prefixes, assigning high process scores if rollout pass rates are high; during test-time inference, PRMs serve as heuristic value evaluators guiding Tree Search and Best-of-$N$ pruning.

Multi-step mathematical theorem proving, fine-grained credit assignment in reasoning RL, and tree-search guidance.

Sparse terminal rewards cause massive credit assignment variance in long-horizon reasoning; PRMs inject dense, step-level supervision signals to guide flawless logical deductions.

Contrast Outcome Supervision (ORM) vs Process Supervision (PRM) in eliminating lucky guesses and hallucinated intermediate deductions?

Explain how Math-Shepherd constructs automated step-level PRM datasets using Monte Carlo tree rollouts without human annotation?

Why does the product of step-wise PRM probabilities outperform terminal scores in Best-of-$N$ trajectory selection?

Why do large-scale search algorithms ($N > 1000$) exploit reward hacking vulnerabilities in learned PRMs and how to regularize them?

How to integrate step-level PRM dense rewards into PPO/GRPO policy gradients to eliminate temporal variance in credit assignment?