Process Reward Model

A PRM (Process Reward Model) scores each intermediate reasoning step, refining credit assignment from 'was the answer right' to 'which step went wrong': outcome reward models (ORMs) give one overall 0/1 and cannot tell the policy which step to fix, while step-level rewards deliver gradient signal to the exact step. Labels come from human step annotation or Monte-Carlo estimation (MC-EST): sample multiple rollouts from a step and average their terminal outcomes as that step's quality — the Math-Shepherd approach.

RL perspective: PRMs complement RLVR outcome rewards — RLVR gives cheap, unbiased, unhackable terminal signals, PRMs give fine-grained process signals; at inference time a PRM serves as the node-value function for MCTS (see this module's mcts card) or reranks trajectories in best-of-N. See the LLM module: full definition, formulas, and exam points in the LLM module's prmRlvr card (id: prm-rlvr).

reasoning-model training (step-level reward), test-time search (MCTS node values, step reranking), error localization in multi-step tasks.

sparse outcome rewards make credit assignment impossible — the policy only knows overall right/wrong, so the RL signal is nearly random; PRMs distribute reward to individual steps and, combined with search, markedly raise solution rates on hard reasoning tasks.

Test Your Knowledge: Practice Questions for "Process Reward Model"

Single choice pitfall questions with instant feedback and mistake tracking.