RL with Verifiable Rewards

RLVR (Reinforcement Learning from Verifiable Rewards) uses rule/program-verifiable signals as rewards, replacing human or AI reward models: exact-answer matching for math (parsed to canonical form); unit tests for code, with reward $pass@k$ — probability that among $k$ sampled programs at least one passes all tests; ground-truth comparison for logic/multiple-choice. Such rewards cost nothing to annotate, carry no model bias, and since the verifier is deterministic with no exploitable loophole, they are unhackable.

Pairing with GRPO: RLVR is GRPO's natural match — sample $G$ responses $\{o_1, \dots, o_G\}$ per prompt $q$, score each with the verifier ($r_i$), normalize within the group to get advantage $A_i = \frac{r_i - \text{mean}(r_1, \dots, r_G)}{\text{std}(r_1, \dots, r_G)}$, then update with ratio clipping — no reward model and no critic anywhere; DeepSeek-R1 is the canonical "RLVR + GRPO" pipeline.

tasks with ground-truth verification (math, code, logic); the primary reward source for post-training o1/R1-style reasoning models, complemented by process reward models (PRM) for step-level signals.

RLHF reward models are expensive, noisy, and hackable; RLVR gives zero-cost, unbiased, deterministic rewards that scale reasoning RL — at the cost of requiring verifiability, so open-ended tasks still need reward models or PRMs.

Test Your Knowledge: Practice Questions for "RL with Verifiable Rewards"

Single choice pitfall questions with instant feedback and mistake tracking.