Offline-to-Online Fine-Tuning

Offline-to-online RL first trains a conservative policy on fixed offline data, then keeps optimizing through online interaction; two key mechanisms: (1) conservatism annealing — gradually decay the offline conservative coefficient (e.g., CQL's $\alpha$ toward 0) so 'conservative inertia' does not suppress online exploration; (2) replay mixing — sample online and offline data in a blended ratio, anchoring on old data to keep distribution shift from resurfacing and training stable.

systems that cold-start from logs then keep interacting — recommenders/ads pretrained on logs before self-adaptation, robots trained in simulation or on history before deployment, LLM alignment pipelines.

tackles the two failure modes of deploying a purely offline policy — suboptimality from residual conservatism (needs loosening as environment signals arrive) and policy collapse during early exploration (old-data replay acts as a safety anchor); it also maps naturally onto RLHF — the SFT/preference data of RLHF stage 1 is an offline dataset, and PPO/GRPO start from the reference policy with a KL anchor, i.e., 'offline data to online optimization' realized in LLMs.

Test Your Knowledge: Practice Questions for "Offline-to-Online Fine-Tuning"

Single choice pitfall questions with instant feedback and mistake tracking.