Policy Gradient Theorem

The policy gradient theorem gives a model-free gradient of $J(\theta)=\mathbb{E}_{\tau}[R(\tau)]$; the core is the log-derivative trick and the expectation form $\nabla_\theta J = \mathbb{E}_{\tau}[\sum_t \nabla_\theta \log \pi_\theta(a_t|s_t) G_t]$. Derivation:

1. Differentiate and multiply/divide by $\pi_\theta(\tau)$: $\nabla_\theta J = \mathbb{E}_{\tau}[\nabla_\theta \log \pi_\theta(\tau) R(\tau)]$, via the log-derivative trick $\nabla_\theta \log \pi_\theta(a|s) = \frac{\nabla_\theta \pi_\theta(a|s)}{\pi_\theta(a|s)}$ (turning a gradient of probabilities into the gradient of their logs, keeping the estimator unbiased); 2. Factor the trajectory $\pi_\theta(\tau) = \mu(s_0)\prod_t \pi_\theta(a_t|s_t)P(s_{t+1}|s_t,a_t)$ and take the log: the dynamics terms $\mu, P$ do not depend on $\theta$ and cancel, leaving $\sum_t \nabla_\theta \log \pi_\theta(a_t|s_t)$ — which is why no model is needed; 3. By causality, action $a_t$ affects only future returns, so replace the full return with reward-to-go $G_t = \sum_{k\ge t}\gamma^{k-t}r_k$ (lower variance), giving the REINFORCE update $\theta \leftarrow \theta + \eta \sum_t \nabla_\theta \log \pi_\theta(a_t|s_t) G_t$.

episodic tasks (full trajectories, MC-style) and continuous action spaces; REINFORCE is the simplest implementation, and GRPO/RLHF group-relative updates build on it.

differentiating the probability directly is hard and full-return estimates have huge variance. Subtracting a baseline $b(s)$ gives $\nabla_\theta J = \mathbb{E}[\nabla_\theta \log \pi_\theta(a|s)(G_t - b(s))]$. Unbiasedness proof: $\mathbb{E}[\nabla_\theta \log \pi_\theta(a|s) b(s)] = b(s)\sum_a \pi_\theta(a|s)\frac{\nabla_\theta \pi_\theta(a|s)}{\pi_\theta(a|s)} = b(s)\nabla_\theta\sum_a \pi_\theta(a|s) = b(s)\nabla_\theta 1 = 0$, so a baseline never biases the gradient and only cancels action-independent variance; $b(s)=V(s)$ gives the advantage form and leads to Actor-Critic; step-size sensitivity led to TRPO/PPO.

Whiteboard: derive the policy gradient theorem from J(θ)=E_τ[R(τ)]; why do the dynamics terms cancel?

What is the log-derivative trick, and why does it yield an unbiased gradient estimator?

Prove E[∇log π(a|s)·b(s)] = 0 and explain why a baseline reduces variance without biasing the gradient.

Why does reward-to-go G_t have lower variance than the full return? How does REINFORCE relate to MC methods?

Root causes of high variance and step-size sensitivity; which improvements (TRPO/PPO/GAE) emerged?