Group Relative Policy Optimization

GRPO (Group Relative Policy Optimization) is the online policy-optimization algorithm used by DeepSeekMath/R1 for RLVR-style reasoning RL. Three steps: (1) sample $G$ responses $\{o_1, \dots, o_G\}$ per prompt $q$; (2) score each with the verifier and compute a group-relative advantage: $$A_i = \frac{r_i - \text{mean}(r_1, \dots, r_G)}{\text{std}(r_1, \dots, r_G)}$$ subtracting the group mean and dividing by the group std — within a group, $A_i$ has zero mean and unit variance, so the advantage depends only on relative ranking, not absolute reward scale; this is the key to dropping the critic. (3) plug $A_i$ into a PPO-style clipped surrogate with a KL term: $$\mathcal{J}_{GRPO}(\theta) = \mathbb{E}\left[ \frac{1}{G} \sum_{i=1}^{G} \min\left( \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{old}}(o_i|q)} A_i, \; \operatorname{clip}\left( \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{old}}(o_i|q)}, \, 1-\varepsilon, \, 1+\varepsilon \right) A_i \right) - \beta\, D_{KL}(\pi_\theta \| \pi_{ref}) \right]$$ The ratio $\pi_\theta(o_i|q)/\pi_{\theta_{old}}(o_i|q)$ measures how much the policy moved on response $o_i$; $\min$ plus clip ($\varepsilon \approx 0.2$) bounds the per-update step; $\beta D_{KL}(\pi_\theta \| \pi_{ref})$ penalizes deviation from the reference policy $\pi_{ref}$, in practice estimated with the non-normalized KL $D_{KL} \approx \mathbb{E}\left[ \frac{\pi_{ref}(o|q)}{\pi_\theta(o|q)} - \log\frac{\pi_{ref}(o|q)}{\pi_\theta(o|q)} - 1 \right]$ — lower variance and always non-negative.

Why dropping the critic saves VRAM: PPO estimates advantages with GAE $A_t = \delta_t + \gamma\lambda\delta_{t+1} + \gamma^2\lambda^2\delta_{t+2} + \dots$, where the TD error $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ needs a value network $V_\phi$ — same size as the policy, so its parameters, gradients, and optimizer states (Adam moments) cost roughly as much VRAM as training another large model, which is nearly infeasible at 100B+ scale. GRPO replaces the whole value network with one sampling pass plus group statistics, eliminating the critic's forward/backward and the GAE expansion — training memory drops by roughly one model-scale's worth, which is exactly why DeepSeek-V3/R1 can run full-parameter RL on a 671B MoE. Group normalization also means absolute reward-scale drift (e.g. 0.5 → 0.8) does not change ranking, so $A_i$ is unchanged and training is scale-robust.

reasoning RL on verifiable tasks (math, code) via RLVR; especially suited to sparse rewards needing heavy exploration; the standard post-training paradigm for o1/R1-style reasoning models, also used for general dialogue optimization.

PPO+critic memory/compute bottlenecks and instability from reward-scale drift; GRPO fixes both via group-relative normalization plus an explicit KL regularizer against policy drift and collapse. Complementary to DPO: GRPO needs online sampling and a verifier (RLVR); DPO is offline, for static preference data.

Write GRPO's group-relative advantage formula and explain why group normalization removes the critic?

Write GRPO's full objective (ratio, clip, KL term) and explain every symbol?

Why is GRPO more memory-efficient than PPO? Compare memory footprints (critic/GAE)?

Role of the KL penalty? Why the non-normalized KL estimator? How does group normalization resist reward-scale drift?

GRPO vs PPO vs DPO: use cases and selection criteria?