PPO Clipped Objective

PPO (Proximal Policy Optimization) approximates TRPO's KL trust region with a clipped probability ratio, needing only first-order optimization. With the ratio $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ (an importance-sampling weight measuring the relative probability of the state-action pair under new vs old policy), the clipped objective is

$\mathcal{L}^{CLIP}(\theta) = \mathbb{E}_t\left[\min\left(r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta), 1-\varepsilon, 1+\varepsilon)\hat{A}_t\right)\right],\quad \varepsilon = 0.2$ \nDerivation: 1. Start from the policy-gradient surrogate $L^{PG}(\theta) = \mathbb{E}_t[r_t(\theta)\hat{A}_t]$: unconstrained maximization lets $r_t$ grow unboundedly and one large update collapses the policy (TRPO's motivation); 2. Add clipping: once $r_t$ leaves $[1-\varepsilon, 1+\varepsilon]$ it is clamped, and taking the min with the unclipped term makes the objective constant in $r_t$ (zero gradient) beyond the region — a one-line trust region; 3. Numeric example ($\varepsilon = 0.2$): with $\hat{A}_t = +1, r_t = 2$, $\min(2, 1.2) = 1.2$ — the positive advantage caps the ratio at 1.2; with $\hat{A}_t = -1, r_t = 0.5$, $\min(-0.5, -0.8) = -0.8$ — the negative advantage floors it at 0.8 (clipping in the opposite direction). The per-step ratio drift is bounded to about $\pm 20\%$, equivalent to a small KL trust region.

policy optimization in RLHF (with KL penalty and GAE), RLVR/reasoning RL; OpenAI's default general-purpose RL algorithm for both continuous and discrete actions and large-scale distributed training.

REINFORCE is step-size sensitive and TRPO needs expensive second-order optimization. PPO pins the update inside a trust region with min+clip so plain first-order SGD trains stably — simple, robust, reproducible, and the de facto RLHF standard.

Write PPO's clipped objective; what is the default ε, and why the min?

Whiteboard: with A=+1,r=2 and A=−1,r=0.5 show numerically how min+clip bounds the update.

What does the ratio r_t(θ) mean, and what role does importance sampling play?

Why do positive and negative advantages clip in opposite directions; when is the clip active?

How does PPO relate to TRPO, and why is PPO the default for RLHF/RLVR?