F4Medium★ Core Essential · Industrial BedrockPart F · Alignment & Reinforcement Learning

Group Relative Policy Optimization (GRPO)

⚡Flashcard Drill➔

Critic-free policy optimization normalizing rewards within a sampled response group per prompt.

⏱️ Time: O(B * G) 极轻量纯向量运算
💾 Space: 相比 PPO 节省整整一套 Critic 网络的百亿显存
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def compute_grpo_advantages(rewards: np.ndarray, eps: float = 1e-8) -> np.ndarray:
"""
计算 GRPO 组内相对优势。
参数:
rewards: (B, G) 每个 Prompt 生成 G 个采样的奖励标量
返回:
advantages: (B, G) 归一化后的组内相对优势
"""
# 沿组维度 (axis=-1) 独立计算均值与标准差
mean =
std = np.std(rewards, axis=-1, keepdims=True)
# 组内相对归一化
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What rule-based rewards were used in DeepSeek-R1 Zero-RL with GRPO?
Next: Generalized Advantage Estimation (GAE) →
💡

Core Mental Anchor / Mnemonic

Sample G completions per prompt, z-score rewards within the group, and drop the Critic network.

📐 Mathematical Derivation & Core Formula

A^i,j=ri,j−mean({ri,⋅})std({ri,⋅})+ϵ\hat{A}_{i, j} = \frac{r_{i, j} - \mathrm{mean}(\{r_{i, \cdot}\})}{\mathrm{std}(\{r_{i, \cdot}\}) + \epsilon}
### Critic-Free Group-Relative Advantage Normalization
GRPO removes the memory-heavy Critic network required by PPO:
1. Sample a group of GG candidate completions {o1,…,oG}\{o_1, \dots, o_G\} for each prompt qq;
2. Score each completion with rule-based verifiers or a reward model to obtain {r1,…,rG}\{r_1, \dots, r_G\};
3. Compute group-standardized relative advantages:
A^i=ri−μrσr+ϵ\hat{A}_i = \frac{r_i - \mu_r}{\sigma_r + \epsilon}
4. Apply PPO-style clipping and KL regularization against the reference policy.

🔄 Tensor Dimensions & Shape Flow

rewards: (B, G) -> group mean, std -> (B, 1) -> (rewards - mean)/(std + eps) -> advantages: (B, G)

🛡️ Numerical Stability & Pitfalls Checklist

  • Use adequate group size G (e.g., 8–64) so intra-group mean and std estimates are not too noisy.
  • Add eps (1e-8) in denominator for degenerate groups where all responses receive identical rewards.
  • DeepSeek GRPO pairs group advantages with a non-negative unbiased KL estimator against reference.

💻 Reference Code Implementation

import numpy as np

def compute_grpo_advantages(rewards: np.ndarray, eps: float = 1e-8) -> np.ndarray:
    """
    计算 GRPO 组内相对优势。
    参数:
        rewards: (B, G) 每个 Prompt 生成 G 个采样的奖励标量
    返回:
        advantages: (B, G) 归一化后的组内相对优势
    """
    # 沿组维度 (axis=-1) 独立计算均值与标准差
    mean = np.mean(rewards, axis=-1, keepdims=True)
    std = np.std(rewards, axis=-1, keepdims=True)
    
    # 组内相对归一化
    return (rewards - mean) / (std + eps)

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
# 模拟一个 Prompt 的 4 个生成,得分分别为 [1.0, 0.0, 1.0, 0.0]
rewards = np.array([[1.0, 0.0, 1.0, 0.0]])
advs = compute_grpo_advantages(rewards)
# 均值应为 0,且好回答优势为正,差回答优势为负
assert np.isclose(advs.mean(), 0.0)
assert advs[0, 0] > 0 and advs[0, 1] < 0
assert np.isclose(advs[0, 0], -advs[0, 1])
print("✓ GRPO 组内优势归一化自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:What rule-based rewards were used in DeepSeek-R1 Zero-RL with GRPO?
Deterministic accuracy rewards (checking final math answers or passing compiler/unit tests for code) combined with format rewards enforcing structured reasoning tags (such as <think>...</think>), avoiding neural reward-model hacking.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: PPO Clipped Surrogate Objective LossBack to Part F & All 105Next: Generalized Advantage Estimation (GAE) →