Actor-Critic

Actor-Critic combines an Actor (policy network $\pi_\theta(a|s)$ generating actions) and a Critic (value network $V_w(s)$ evaluating states): the Actor updates along the policy-gradient direction, and the Critic supplies a value baseline and advantage estimates to reduce variance. Core formulas:

$\nabla_\theta J = \mathbb{E}_{s,a}\left[\nabla_\theta \log \pi_\theta(a|s) A^{\pi}(s,a)\right],\quad A(s,a) = Q^{\pi}(s,a) - V^{\pi}(s)$ \nDerivation: 1. Start from the policy gradient theorem and substitute $Q^{\pi}(s,a) = r + \gamma V^{\pi}(s')$ into the advantage: $\nabla_\theta J = \mathbb{E}[\nabla_\theta \log \pi_\theta(a|s)\underbrace{(r + \gamma V_w(s') - V_w(s))}_{\delta_{TD}}]$ — a one-step bootstrapped TD error serves as the advantage estimate; 2. The Critic fits the value by least squares: $L(w) = \mathbb{E}[(r + \gamma V_w(s') - V_w(s))^2]$ with $\nabla_w L = -2\mathbb{E}[\delta_{TD}\nabla_w V_w(s)]$ (TD(0): biased but low variance due to bootstrapping); 3. Generalize to TD($\lambda$)/GAE: $A_t = \sum_{l=0}^{\infty}(\gamma\lambda)^l\delta_{t+l}$ with $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$. $\lambda=0$ is pure TD advantage (low variance, high bias), $\lambda=1$ is MC advantage (high variance, low bias); GAE is the standard in PPO/RLHF; 4. The Actor updates: $\theta \leftarrow \theta + \eta \nabla_\theta \log \pi_\theta(a_t|s_t)\hat{A}_t$. The baseline is unbiased as in policy gradients: $\mathbb{E}[\nabla_\theta \log \pi \cdot V(s)] = 0$ — it only removes variance.

the shared backbone of on- and off-policy algorithms — A2C/A3C, PPO, SAC, TD3, and GRPO (which replaces the Critic with group-relative rewards) are all Actor-Critic variants.

REINFORCE uses full returns with high variance and learns no value. The Critic's $V(s)$ baseline cancels state-dependent variance without biasing the gradient and provides credit assignment (the advantage $A(s,a)$ scores each action relative to its state); Actor and Critic alternately supervise each other — the most important deep-RL framework.

Define the advantage A(s,a)=Q(s,a)−V(s) and justify using it over Q or the raw reward.

Whiteboard: derive the Actor's TD-error update from the policy gradient theorem and prove the V(s) baseline is unbiased.

How is the Critic loss derived? How do TD(λ)/GAE trade off bias and variance?

How does Actor-Critic improve on REINFORCE? Why lower variance and online training?

How do A2C/A3C, PPO, SAC, GRPO reuse the Actor-Critic backbone, and what do they change?