Offline Distribution Shift

Distribution shift is the core difficulty of offline RL: the value function and policy learn only from a fixed dataset $\mathcal{D}$, yet during training the policy keeps querying Q on out-of-distribution (OOD) state-action pairs the data never covered; with no ground-truth supervision for these actions, Q must extrapolate, yielding systematically inflated estimates (extrapolation error).

domains with historical logs but no safe online interaction — recommender systems, ads, autonomous driving, healthcare, robotics; also the hidden hazard when RLHF trains on offline preference data. Online RL escapes it because environment feedback corrects errors in real time.

it explains why naive offline Q-learning must fail — extrapolation error is amplified by Bellman bootstrapping (bootstrapping error amplification): updating $Q(s,a) \leftarrow r + \gamma \max_{a'} \hat{Q}(s',a')$, the $\max$ operator selects the most erroneous $a'$, so the error propagates round after round, Q inflates globally, and the policy drifts toward OOD actions; hence conservative offline algorithms (CQL, IQL, BCQ).

What is the mechanism of extrapolation error? Why do Q-estimates for OOD actions inflate systematically?

How does Bellman bootstrapping amplify extrapolation error? Write the error-propagation chain in $Q(s,a)=r+\gamma \max_{a'} \hat{Q}(s',a')$ and explain the max operator's role.

How does offline distribution shift differ from, and relate to, online DQN overestimation?

Why must an offline policy stay in-distribution? Enumerate three mitigation families (policy constraint, conservative Q, data generation) with one algorithm each.

Given a poorly covered fixed dataset, how would you diagnose extrapolation error from Q-values, and what training metric warns of policy collapse?