Offline-to-Online Fine-Tuning

Offline-to-online RL first trains a conservative policy on fixed offline data, then keeps optimizing through online interaction; two key mechanisms: (1) conservatism annealing — gradually decay the offline conservative coefficient (e.g., CQL's $\alpha$ toward 0) so 'conservative inertia' does not suppress online exploration; (2) replay mixing — sample online and offline data in a blended ratio, anchoring on old data to keep distribution shift from resurfacing and training stable.

systems that cold-start from logs then keep interacting — recommenders/ads pretrained on logs before self-adaptation, robots trained in simulation or on history before deployment, LLM alignment pipelines.

tackles the two failure modes of deploying a purely offline policy — suboptimality from residual conservatism (needs loosening as environment signals arrive) and policy collapse during early exploration (old-data replay acts as a safety anchor); it also maps naturally onto RLHF — the SFT/preference data of RLHF stage 1 is an offline dataset, and PPO/GRPO start from the reference policy with a KL anchor, i.e., 'offline data to online optimization' realized in LLMs.

Why must conservatism be annealed in offline-to-online? What happens if you go online with a large penalty coefficient?

What is the role of replay mixing? How does the new/old data sampling ratio affect stability?

CQL or IQL for offline-to-online? Why is IQL often more stable?

How does RLHF relate to offline-to-online? How does the reference-policy KL constraint act as 'conservatism'?

How do you schedule the anneal from conservative to fully online? When can old-data replay be dropped?