TRPO → PPO Lineage

TRPO (Trust Region Policy Optimization) constrains the single-step policy update via a KL trust region to guarantee monotonic improvement:

$\max_\theta\; \mathbb{E}_{s,a\sim\pi_{\theta_{old}}}\left[\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)} A^{\pi_{\theta_{old}}}(s,a)\right] \quad \text{s.t.}\; \mathbb{E}_s\left[D_{KL}(\pi_{\theta_{old}}(\cdot|s) \| \pi_\theta(\cdot|s))\right] \le \delta$ \nDerivation: 1. Linearize the surrogate objective at $\theta_{old}$: maximize $g^T\Delta\theta$ with $g = \nabla_\theta\mathbb{E}_{s,a\sim\pi_{\theta_{old}}}[\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)} A]$ — the expectation is sampled under the old policy (the probability ratio is the importance weight); 2. The KL constraint vanishes to first order at $\theta_{old}$; expanding to second order gives $\frac{1}{2}\Delta\theta^T F\Delta\theta \le \delta$ with the Fisher matrix $F = \mathbb{E}_{s,a}[\nabla_\theta\log\pi_\theta(a|s)\nabla_\theta\log\pi_\theta(a|s)^T]$ (negative expected Hessian — the Hessian approximation, no second-order derivatives needed); 3. Lagrangian solution gives the natural gradient $\Delta\theta = \alpha F^{-1}g$; solve $Fx = g$ by conjugate gradients (avoiding the $O(n^2)$ explicit inverse) and backtrack the step $\alpha$ by line search using the monotonic-improvement bound (KKL, with $C = \frac{4\varepsilon\gamma}{(1-\gamma)^2}$).

continuous control where monotonic improvement matters and samples are expensive; the conceptual source of constrained updates in offline RL.

vanilla policy gradient can collapse the policy with one large step. TRPO provides a trust region via a second-order KL constraint, but each iteration needs conjugate gradient and line search — expensive and complex. PPO approximates the same constraint to first order by clipping the ratio to $[1-\varepsilon, 1+\varepsilon]$ (a small KL trust region), keeping TRPO's stability with plain first-order SGD — that is the TRPO → PPO lineage: constrained optimization becomes a clipped surrogate objective.

Write TRPO's constrained problem: the surrogate objective and the KL constraint E[D_KL]≤δ.

Whiteboard: linearize the objective, expand KL to second order, derive the natural gradient Δθ=αF⁻¹g, and explain why the Fisher matrix approximates the Hessian.

How does TRPO guarantee monotonic improvement? Roles of line search and the coefficient C=4εγ/(1-γ)².

The TRPO → PPO lineage: how does ratio clipping approximate the KL trust region?

Where is TRPO's computational bottleneck (Hessian/conjugate gradient), and how is it accelerated in practice?