PPO Clipped Objective

PPO (Proximal Policy Optimization) constrains policy updates via a clipped probability ratio. With the ratio $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$, the clipped objective is $L^{CLIP}(\theta) = \mathbb{E}_t[\min(r_t(\theta) \hat{A}_t,\; \operatorname{clip}(r_t(\theta), 1-\varepsilon, 1+\varepsilon) \hat{A}_t)]$ with $\varepsilon = 0.2$: r is capped at 1.2 for positive advantages and floored at 0.8 for negative ones — e.g. $r=2$ is clipped to $1.2$, preventing policy jumps.

policy optimization in the RLHF stage (with KL penalty and GAE); also OpenAI's default general-purpose RL algorithm for large-scale distributed training over continuous and discrete action spaces.

vanilla policy gradient (REINFORCE) is hypersensitive to step size — one large update collapses the policy; TRPO guarantees monotonic improvement but needs expensive second-order optimization. PPO approximates the constraint with clipping, achieving stable first-order training simply and reliably.

Write the PPO clipped objective and explain the clip's role. Why ε = 0.2?

How does clipping behave for positive vs negative advantages? What happens at r=2?

What does the ratio r_t(θ) mean, and what role does importance sampling play?

How does PPO relate to TRPO, and why is PPO preferred in practice?

How is the reward constructed in RLHF-PPO? Roles of GAE and the KL penalty?