SAC Max-Entropy

SAC (Soft Actor-Critic) is a maximum-entropy off-policy Actor-Critic whose objective augments cumulative reward with a policy entropy regularizer:

$\mathcal{J}(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_t \gamma^t\left(r(s_t,a_t) + \alpha\mathcal{H}(\pi(\cdot|s_t))\right)\right],\quad \mathcal{H}(\pi(\cdot|s)) = -\mathbb{E}_{a\sim\pi}[\log \pi(a|s)]$ \nDerivation: 1. Soft policy evaluation: the entropy bonus turns the Bellman target into $y = r + \gamma\left(Q_{\theta^-}(s',a') - \alpha\log\pi_\phi(a'|s')\right)$ with $a'\sim\pi_\phi$ (subtracting the entropy term from the value); the Critic minimizes $L(\theta_i) = \mathbb{E}[(y - Q_{\theta_i}(s,a))^2]$; 2. Soft policy improvement: solving $\pi_{new} = \arg\max_\pi \mathbb{E}_{a\sim\pi}[Q^{\pi_{old}}(s,a)] + \alpha\mathcal{H}(\pi(\cdot|s))$ yields the closed-form Boltzmann policy $\pi^*(a|s) \propto \exp(Q(s,a)/\alpha)$ (softmax over value with temperature $\alpha$); in practice a reparameterized Gaussian $a = \mu_\phi(s) + \sigma_\phi(s)\odot\epsilon$ gives differentiable samples minimizing $J(\phi) = \mathbb{E}_{\epsilon}[-Q(s,\tilde{a}) + \alpha\log\pi_\phi(\tilde{a}|s)]$; 3. Twin-Q min: two Critic networks regress the same target whose value uses $\min_i Q_{\theta_i^-}(s',a')$ to suppress max overestimation (a more direct off-policy fix than Double DQN); 4. Auto temperature: treating $\alpha$ as the dual variable of the constraint $\mathcal{H}(\pi) \ge \mathcal{H}_{target}$, the loss is $J(\alpha) = \mathbb{E}_{a\sim\pi}[-\alpha\log\pi(a|s) - \alpha\mathcal{H}_{target}]$ with $\nabla_\alpha J = -\mathbb{E}[\log\pi(a|s) + \mathcal{H}_{target}]$: when measured entropy falls below the target, $\alpha$ increases (more exploration); above it, $\alpha$ decreases.

the de facto SOTA baseline for continuous control (robotics, simulators); scenarios needing stochastic policies (multimodal optimal actions) and high sample efficiency.

deterministic policy gradients (DDPG/TD3) can stick to local optima and are hyperparameter-sensitive. SAC's entropy regularizer encourages exploration and resists local optima and multimodal actions; off-policy learning with replay gives sample efficiency, and twin-Q with auto temperature makes training robust — the default for continuous control.

Write SAC's objective and explain the meaning of maximum entropy.

Whiteboard: derive the soft Bellman equation and the closed-form improvement π∝exp(Q/α) from the entropy-regularized objective.

What does twin-Q min solve, and how does it differ from Double DQN?

Design of the auto-temperature loss J(α); does α increase or decrease when entropy is below the target?

Why is SAC the default for continuous control? Benefits of off-policy + stochastic policy + entropy?