Reference-Free Alignment SimPO/ORPO/KTO

reference-free alignment methods drop the reference policy and/or the separate SFT stage: ① SimPO (Simple Preference Optimization) removes the reference model entirely, using the sequence-level average log-probability $\frac{1}{|y|}\sum_t \log \pi_\theta(y_t|y_{<t}, x)$ as an implicit reward (vs DPO's same-size reference model, typically β = 0.1), saving ~50% VRAM and training time; ② ORPO (Odds Ratio Preference Optimization) adds a preference term on top of the SFT cross-entropy, doing SFT + alignment in one step; ③ KTO (Kahneman-Tversky Optimization) trains on per-response desirable/undesirable signals, supporting unpaired data.

low-cost fine-tuning in the open-source community (7B on a single GPU), scenarios with only thumbs-up/down feedback, and teams iterating rapidly on alignment experiments.

DPO needs a reference model the size of the policy, doubling memory, and requires paired data. SimPO cuts VRAM ~50%, ORPO merges SFT + alignment into one step, and KTO lifts the paired-data requirement — collectively lowering the engineering bar for alignment.

Test Your Knowledge: Practice Questions for "Reference-Free Alignment SimPO/ORPO/KTO"

Single choice pitfall questions with instant feedback and mistake tracking.