Process Reward Model

A PRM (Process Reward Model) scores each intermediate reasoning step, refining credit assignment from 'was the answer right' to 'which step went wrong': outcome reward models (ORMs) give one overall 0/1 and cannot tell the policy which step to fix, while step-level rewards deliver gradient signal to the exact step. Labels come from human step annotation or Monte-Carlo estimation (MC-EST): sample multiple rollouts from a step and average their terminal outcomes as that step's quality — the Math-Shepherd approach.

RL perspective: PRMs complement RLVR outcome rewards — RLVR gives cheap, unbiased, unhackable terminal signals, PRMs give fine-grained process signals; at inference time a PRM serves as the node-value function for MCTS (see this module's mcts card) or reranks trajectories in best-of-N. See the LLM module: full definition, formulas, and exam points in the LLM module's prmRlvr card (id: prm-rlvr).

reasoning-model training (step-level reward), test-time search (MCTS node values, step reranking), error localization in multi-step tasks.

sparse outcome rewards make credit assignment impossible — the policy only knows overall right/wrong, so the RL signal is nearly random; PRMs distribute reward to individual steps and, combined with search, markedly raise solution rates on hard reasoning tasks.

PRM vs ORM: differences? How does a PRM refine credit assignment?

How are PRM training labels obtained (human step labels / Monte-Carlo MC-EST)?

What role does a PRM play in MCTS and best-of-N reranking?

Main PRM challenges (label noise, error accumulation, generator/verifier coverage mismatch)?

How do RLVR outcome rewards and PRM process rewards complement each other? Trade-offs?