Alignment Tax & Preference Data

Alignment Tax is the phenomenon where alignment training (RLHF/DPO) improves helpfulness/harmlessness but can degrade other capabilities (math, code, long-tail knowledge, output diversity, creativity), quantifiable as relative regression on a benchmark $\text{Tax}(B) = \frac{S_{\text{base}}(B) - S_{\text{aligned}}(B)}{S_{\text{base}}(B)}$; Preference Data — contrastive chosen/rejected pairs from human labelers (or AI judges) — is the core fuel of RLHF/DPO-style methods, and its scale, quality, and distribution set the ceiling of alignment quality. Typical mitigations: tuning the KL coefficient and data mix, multi-objective weighting, and regression checks on capability benchmarks.

choosing alignment methods and strength (KL penalty size, data volume), designing preference-data pipelines (crowd annotation, RLAIF synthesis, labeler-agreement checks), and pre-release capability regression checks (benchmark scores before/after alignment).

alignment is not free — blindly maximizing preference scores causes capability regression that must be measured and mitigated; and garbage-in-garbage-out: noise and bias in preference data propagate directly into the aligned model, capping final quality.

What is alignment tax? Typical manifestations and which capabilities are most affected?

How do you measure alignment tax, and how to mitigate it (KL coefficient, data mix, multi-objective)?

How does preference-data quality affect alignment, and what makes a good preference pair?

How is labeler agreement evaluated, and what does low agreement indicate?

Compare RLAIF (AI feedback) vs human preference data: pros and cons?