Auto Metrics vs LLM-as-Judge

three classic automatic metrics and their limits: (1) BLEU — n-gram precision with a brevity penalty, $\text{BLEU} = BP \cdot \exp\big(\sum_n w_n \log p_n\big)$, built for MT, blind to synonyms and legitimate paraphrases; (2) ROUGE — recall-oriented n-gram / LCS F-measure for summarization, also ignoring semantics; (3) perplexity $\text{PPL}(x) = \exp\big(-\tfrac{1}{N} \sum_i \log P(x_i \mid x_{<i})\big)$ — measures fit, not quality; favors short text and misses factual errors. LLM-as-judge uses a large model as scorer/comparator, with agreement measured by Cohen's Kappa: $\kappa = \tfrac{p_o - p_e}{1 - p_e}$ (observed agreement minus chance agreement).

cheap regression testing for summarization/translation/open-ended generation; LLM-as-judge for subjective quality, long text, or tasks without gold references; validating judge reliability with Kappa (judge-vs-human or judge-vs-judge) before trusting automated verdicts.

classic metrics score surface similarity, not semantic correctness — high scores can hide mistranslations or missing points, and perplexity measures fit, not quality. LLM-as-judge brings semantic judgments but introduces systematic biases — position bias (preferring later answers), self-preference, verbosity bias — so Kappa and debiasing designs (swapped order) are required to trust it.

Test Your Knowledge: Practice Questions for "Auto Metrics vs LLM-as-Judge"

Single choice pitfall questions with instant feedback and mistake tracking.