Conservative Q-Learning

Conservative Q-Learning (CQL) is a SOTA offline RL baseline that adds a conservative regularizer to the standard Bellman update, systematically suppressing Q-estimates of OOD actions: $\min_Q \alpha \left( \mathbb{E}_{a \sim \pi_\beta} \left[ \log \sum_{a'} \exp Q(s,a') \right] - \mathbb{E}_{(s,a) \sim \mathcal{D}}[Q(s,a)] \right) + \text{Bellman}$ — the log-sum-exp term pushes up Q for all actions (soft-max approximating $\max_{a'}$), while the data-expectation term pushes up Q only on in-dataset $(s,a)$; the difference systematically lowers OOD actions while keeping in-data Q truthful.

offline decisions without online interaction — recommender/ads logs, safety-critical robot and driving control; also a common initialization for offline-to-online fine-tuning.

provable conservatism — the learned $\hat{Q}^\pi$ is a lower bound on the true $Q^\pi$ (pointwise $\hat{Q}^\pi(s,a) \le Q^\pi(s,a)$): the regularizer injects a conservative offset per state and Bellman iterations preserve the bound, so policy optimization never chases inflated OOD Q-values — extrapolation error is severed at the value level.

Why does CQL yield a Q lower bound? Write the CQL objective and give the derivation intuition for $\hat{Q}^\pi \le Q^\pi$.

What do the log-sum-exp term and the data-expectation term do? Why soft-max instead of max?

CQL vs policy-constraint methods (BCQ / TD3+BC): conservative value vs constrained policy?

How does CQL's coefficient $\alpha$ trade off conservatism vs optimality? Consequences of too large or too small?

How does CQL's Q lower bound serve offline-to-online fine-tuning?