RLHF 3-Stage Pipeline

RLHF (Reinforcement Learning from Human Feedback) is the 3-stage pipeline for aligning LLMs to human preferences: ① SFT instruction tuning; ② reward model (RM) training on human-labeled preference pairs via the Bradley-Terry model $P(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l))$ — the RM models relative preference, not absolute scores; ③ PPO optimization with a KL-penalized reward $r = r_\phi(x,y) - \beta \log(\pi_\theta(y|x)/\pi_{\text{ref}}(y|x))$.

the final stage of LLM training, aligning pretrained + SFT models to human values (helpfulness, harmlessness, honesty); the standard paradigm pioneered by InstructGPT and adopted by ChatGPT, also used for style and safety alignment.

SFT alone cannot express preference direction (which of two answers is better) and overfits easily; RLHF drives policy exploration with an explicit reward signal, robustly matching human preferences on open-ended generation.

Walk through the 3 stages of RLHF: inputs, outputs, and objectives of each stage?

How is the reward model trained? The Bradley-Terry loss and training data format?

Why is the KL penalty βlog(πθ/πref) needed in the PPO stage? What happens without it?

Why does the RM model preference rather than absolute scores? What is reward hacking?

Compare RLHF vs DPO: pipeline, cost, and stability differences?