← Back to Reinforcement Learning Mind Map
🎮 Reinforcement LearningID: rlvr

RL with Verifiable Rewards

RLVR 可验证奖励
🎯Core Definition
RLVR (Reinforcement Learning from Verifiable Rewards) uses rule/program-verifiable signals as rewards, replacing human or AI reward models: exact-answer matching for math (parsed to canonical form); unit tests for code, with reward pass@kpass@k — probability that among kk sampled programs at least one passes all tests; ground-truth comparison for logic/multiple-choice. Such rewards cost nothing to annotate, carry no model bias, and since the verifier is deterministic with no exploitable loophole, they are unhackable.
📌Overview
Pairing with GRPO: RLVR is GRPO's natural match — sample GG responses {o1,…,oG}\{o_1, \dots, o_G\} per prompt qq, score each with the verifier (rir_i), normalize within the group to get advantage Ai=ri−mean(r1,…,rG)std(r1,…,rG)A_i = \frac{r_i - \text{mean}(r_1, \dots, r_G)}{\text{std}(r_1, \dots, r_G)}, then update with ratio clipping — no reward model and no critic anywhere; DeepSeek-R1 is the canonical "RLVR + GRPO" pipeline.
💡Use Cases
tasks with ground-truth verification (math, code, logic); the primary reward source for post-training o1/R1-style reasoning models, complemented by process reward models (PRM) for step-level signals.
⚡Key Problems Solved
RLHF reward models are expensive, noisy, and hackable; RLVR gives zero-cost, unbiased, deterministic rewards that scale reasoning RL — at the cost of requiring verifiability, so open-ended tasks still need reward models or PRMs.
🎯Core Interview Exam Points
Fundamentals · Production Scenarios · Deep Dive
Core Def
Explain the core definition, underlying mechanism/math, and key principles of RL with Verifiable Rewards.
Use Cases
In production engineering, what are the primary use cases for RL with Verifiable Rewards and what core pain points does it solve?
🔥 High-Frequency Deep-Dive Questions
1
Types of verifiable rewards in RLVR? Why is RLVR considered unhackable?
2
How do RLVR and GRPO combine (G samples, group-relative advantage, no critic)?
3
How is the pass@k reward defined for code? How does it differ from answer matching?
4
Which tasks suit RLVR? What verifiability is required, and what about open-ended tasks?
5
RLVR vs reward-model RLHF: compare cost, bias, and robustness?
Updated 2026-08-12
🎯
Verify Mastery: Practice Exam Questions for "RL with Verifiable Rewards"
Take pitfall multiple-choice questions with derivation options; mistakes are automatically logged.
🚀 Start Card Practice ➔
← Previous CardLong-Term Retention RLNext Card →Reward Design & Hacking

🔗 More Reinforcement Learning Knowledge Cards

引用出处: TalentMe 450+ 工业级 AI 架构考点知识库 (Tech Vault) · 遵循 CC-BY-4.0 知识共享协议
URL: https://talentme.airsota.com/resources/tech/roadmap/mindmap/RL/card/rlvr
中文速览与面试要点

RLVR 可验证奖励 (RL with Verifiable Rewards)

定义: RLVR (Reinforcement Learning from Verifiable Rewards, 可验证奖励强化学习) 用程序/规则可自动判定的信号作为奖励,替代人工或 AI 奖励模型:数学任务用答案精确匹配 (解析为规范形式后比较);代码任务用单元测试,奖励取 pass@k——采样 k 个程序、至少一个通过全部测试的概率;逻辑/多选任务用标准答案比对。这类奖励零标注成本、无打分模型偏差,且验证器是确定性的、不存在可被 exploit 的漏洞,因此"不可被黑客攻击" (unhackable)。 与 GRPO 的组合: RLVR 与 GRPO 是天然配对——对每个提示 q 采样 G 个回答 \o_1, , o_G\,验证器逐回答打分 r_i,组内归一化得到 Advantage A_i = r_i - mean(r_1, , r_G) std(r_1, , r_G) 后再做截断策略更新,全程无需奖励模型与 Critic;DeepSeek-R1 正是 "RLVR + GRPO" 的标准化范式。 使用场景: 数学、代码、逻辑等存在 ground-truth 验证的任务;作为 o1/R1 类推理模型后训练的主奖励源,常与过程奖励 (PRM) 互补——RLVR 给终局结果,PRM 给逐步信号。 解决的问题: RLHF 奖励模型训练成本高、打分有噪声且会被 hacking;RLVR 提供零成本、无偏、确定性的奖励信号,使大规模推理强化成为可能;局限是需要可验证的任务——开放式生成任务仍需奖励模型或 PRM 辅助。

核心要点

  • RLVR 的可验证奖励有哪几类?为什么说它是不可被黑客攻击 (unhackable) 的?
  • RLVR 如何与 GRPO 组合 (采样 G 个回答、组内 Advantage、免 Critic)?
  • 代码任务中 pass@k 奖励如何定义?与数学答案匹配有何不同?
  • 哪些任务适合 RLVR?对任务的验证性有什么要求?开放式任务怎么办?
  • RLVR 与奖励模型 RLHF 的对比:成本、偏差、鲁棒性三方面?