Alignment Tax & Preference Data

Alignment Tax is the phenomenon where alignment training (RLHF/DPO) improves helpfulness/harmlessness but can degrade other capabilities (math, code, long-tail knowledge, output diversity, creativity), quantifiable as relative regression on a benchmark $\text{Tax}(B) = \frac{S_{\text{base}}(B) - S_{\text{aligned}}(B)}{S_{\text{base}}(B)}$; Preference Data — contrastive chosen/rejected pairs from human labelers (or AI judges) — is the core fuel of RLHF/DPO-style methods, and its scale, quality, and distribution set the ceiling of alignment quality. Typical mitigations: tuning the KL coefficient and data mix, multi-objective weighting, and regression checks on capability benchmarks.

choosing alignment methods and strength (KL penalty size, data volume), designing preference-data pipelines (crowd annotation, RLAIF synthesis, labeler-agreement checks), and pre-release capability regression checks (benchmark scores before/after alignment).

alignment is not free — blindly maximizing preference scores causes capability regression that must be measured and mitigated; and garbage-in-garbage-out: noise and bias in preference data propagate directly into the aligned model, capping final quality.

Test Your Knowledge: Practice Questions for "Alignment Tax & Preference Data"

Single choice pitfall questions with instant feedback and mistake tracking.