M1Medium★ Core Essential · Eval CalibrationPart M · LLM Eval & RAG/Agent Ops
LLM-as-a-Judge Position Swap & Cohen's Kappa
Mitigate pairwise judge position bias via order swapping and quantify human-judge agreement beyond chance with Cohen's Kappa.
⏱️ Time:
O(N * |Labels|)💾 Space:
O(N)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def resolve_position_swap(vote_ab: str, vote_ba: str) -> str:
"""
Resolve pairwise LLM-as-a-Judge verdicts after swapping candidate order (A, B) -> (B, A).
Votes are 'A', 'B', or 'TIE'. Only consistent winners survive; flipped votes become 'TIE'.
"""
if vote_ab == "A" and vote_ba == "B":
return "A"
if vote_ab == "B" and vote_ba == "A":
return "B"
return "TIE"
def cohens_kappa(rater1: list, rater2: list, labels: list = None) -> float:
"""
Compute Cohen's Kappa agreement between LLM judge and human annotators.
"""
r1 = np.asarray(rater1)
r2 = np.asarray(rater2)
n = len(r1)
if n == 0:
return 0.0
if labels is None:
labels = sorted(set(r1.tolist()) | set(r2.tolist()))
p_o =
p_e = 0.0
for lbl in labels:
p1 = float(np.mean(r1 == lbl))
p2 = float(np.mean(r2 == lbl))
p_e += p1 * p2
if abs(1.0 - p_e) < 1e-12:
return 1.0 if abs(p_o - 1.0) < 1e-12 else 0.0
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Beyond position bias, what other systematic biases affect LLM-as-a-Judge and how do you mitigate them?
Q2: When should you choose Pairwise comparison vs. Pointwise 1-5 scoring?
💡
Core Mental Anchor / Mnemonic
Swap (A, B) to catch slot bias; subtract chance agreement p_e in Kappa and guard (1 - p_e) near zero.
📐 Mathematical Derivation & Core Formula
### Swap Consistency & Chance-Corrected Agreement
LLM judges frequently exhibit position bias, favoring whichever candidate appears first. Evaluating both and reveals ordering bias: if the judge picks the first slot in both runs (
Raw agreement overstates quality under skewed class distributions. Cohen's Kappa adjusts for expected random agreement :
LLM judges frequently exhibit position bias, favoring whichever candidate appears first. Evaluating both and reveals ordering bias: if the judge picks the first slot in both runs (
vote_ab == 'A' and vote_ba == 'A'), the verdicts contradict and resolve to TIE (or escalate).Raw agreement overstates quality under skewed class distributions. Cohen's Kappa adjusts for expected random agreement :
🔄 Tensor Dimensions & Shape Flow
(vote_ab, vote_ba) -> order-swap consistency check -> verdict; (human, judge) -> marginal p_e -> (p_o - p_e)/(1 - p_e) -> kappa
🛡️ Numerical Stability & Pitfalls Checklist
- Flag contradictory swapped pairs (e.g., picking first slot twice) as TIE or ambiguous rather than keeping a single pass.
- Guard the denominator (1 - p_e) against division by zero when both raters collapse onto one label.
- Pairwise swaps double judge token cost; calibrate on a golden validation subset before rolling out to large runs.
💻 Reference Code Implementation
import numpy as np
def resolve_position_swap(vote_ab: str, vote_ba: str) -> str:
"""
Resolve pairwise LLM-as-a-Judge verdicts after swapping candidate order (A, B) -> (B, A).
Votes are 'A', 'B', or 'TIE'. Only consistent winners survive; flipped votes become 'TIE'.
"""
if vote_ab == "A" and vote_ba == "B":
return "A"
if vote_ab == "B" and vote_ba == "A":
return "B"
return "TIE"
def cohens_kappa(rater1: list, rater2: list, labels: list = None) -> float:
"""
Compute Cohen's Kappa agreement between LLM judge and human annotators.
"""
r1 = np.asarray(rater1)
r2 = np.asarray(rater2)
n = len(r1)
if n == 0:
return 0.0
if labels is None:
labels = sorted(set(r1.tolist()) | set(r2.tolist()))
p_o = float(np.mean(r1 == r2))
p_e = 0.0
for lbl in labels:
p1 = float(np.mean(r1 == lbl))
p2 = float(np.mean(r2 == lbl))
p_e += p1 * p2
if abs(1.0 - p_e) < 1e-12:
return 1.0 if abs(p_o - 1.0) < 1e-12 else 0.0
return float((p_o - p_e) / (1.0 - p_e))
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
# 1. Position-bias detection: always picking first slot ('A' then 'A') must resolve to TIE
assert resolve_position_swap("A", "A") == "TIE", "Position bias must degrade to TIE"
assert resolve_position_swap("A", "B") == "A", "Consistent winner A must pass"
assert resolve_position_swap("B", "A") == "B", "Consistent winner B must pass"
# 2. Cohen's Kappa vs random chance agreement
human = ["A", "A", "B", "B", "A", "B"]
judge = ["A", "A", "B", "A", "A", "B"]
k = cohens_kappa(human, judge, labels=["A", "B"])
# Observed agreement = 5/6 = 0.8333, p_e = (3/6)*(4/6) + (3/6)*(2/6) = 0.5 -> kappa = (5/6 - 0.5)/0.5 = 0.6667
assert np.isclose(k, 2.0 / 3.0, atol=1e-4), f"Unexpected kappa: {k}"
print("✓ LLM-Judge swap & Cohen's Kappa assertion passed")🎯 Core Follow-up Inquiries & Reference Answers
Q1:Beyond position bias, what other systematic biases affect LLM-as-a-Judge and how do you mitigate them?
Verbosity bias (favoring longer answers) and self-preference bias (favoring outputs from the same model family). Mitigate via length-controlled bucketing, rubric checklists penalizing filler, and cross-family judge panels calibrated against human Kappa.
Q2:When should you choose Pairwise comparison vs. Pointwise 1-5 scoring?
Pairwise detects subtle quality shifts between two candidate versions better, while Pointwise scales linearly () and works well for continuous longitudinal monitoring when anchored with explicit rubric criteria.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.