Cohen's Kappa Agreement

Cohen's Kappa ($\kappa$) is a statistical metric that measures inter-rater agreement between two evaluators (e.g., an LLM Judge vs Human Expert, or two competing judge models) across qualitative categories, strictly adjusting for chance agreement; the formula is $\kappa = \frac{P_o - P_e}{1 - P_e}$, where $P_o$ represents observed agreement proportion and $P_e$ is the hypothetical expected agreement under chance independence; $\kappa \in [-1, 1]$, with values $>0.8$ denoting near-perfect agreement.

Validating if an automated LLM Judge can safely substitute for human experts, and certifying automated metric alignment against human preferences.

Raw percentage accuracy is heavily distorted under severe class imbalance (e.g., 90% negative baseline); Cohen's Kappa eliminates chance agreement, establishing rigorous statistical credibility for automated LLM evaluation.

Derive formulas for observed agreement $P_o$ and chance agreement $P_e$ from a $2 \times 2$ confusion matrix?

Explain the Landis & Koch benchmark scale and standard industry thresholds (typically $\kappa > 0.7$) for deploying LLM judges in production?

How does Quadratic Weighted Kappa penalize distance severity across ordinal multi-tier ratings (e.g., 1-5 scale)?

Explain the Kappa Paradox (high raw agreement coinciding with low Kappa) under extreme class prevalence skews?

How does Fleiss' Kappa generalize Cohen's Kappa to measure inter-rater reliability across 3+ simultaneous raters?