⚡ LLMsID:
grpoGroup Relative Policy Optimization
GRPO 相对策略优化
🎯Core Definition
GRPO (Group Relative Policy Optimization) is the RL alignment method used by DeepSeek-R1. Core idea: sample responses per prompt and compute advantage from group-relative scores — subtract the group mean, divide by the group std — eliminating the value/critic network entirely; advantage depends only on within-group ranking, not absolute scores.
💡Use Cases
RL for large reasoning models (math, code — tasks with verifiable rewards); standard in DeepSeek-R1/V3, and suited to sparse-reward reasoning RL that needs exploration.
⚡Key Problems Solved
PPO needs a value network the size of the policy, costing VRAM comparable to training another large model; GRPO replaces it with group statistics, cutting training memory substantially and resisting reward-scale drift by construction.
🎯5 High-Frequency Exam Points
1
What's GRPO's key improvement over PPO, and why can it drop the critic?
2
How is GRPO's advantage computed? The group-relative normalization formula?
3
How does R1 combine GRPO with verifiable rewards (RLVR) for reasoning?
4
How does group size G affect training, and how is the KL penalty designed?
5
When would you choose GRPO vs DPO?
📖 In-depth Guide:📄 alignment-and-rlhf-dpo →
Updated 2026-08-11🎯
🚀 Start Card Practice ➔Test Your Knowledge: Practice Questions for "Group Relative Policy Optimization"
Single choice pitfall questions with instant feedback and mistake tracking.