M1Medium★ Core Essential · Eval CalibrationPart M · LLM Eval & RAG/Agent Ops

LLM-as-a-Judge Position Swap & Cohen's Kappa

⚡Flashcard Drill➔

Mitigate pairwise judge position bias via order swapping and quantify human-judge agreement beyond chance with Cohen's Kappa.

⏱️ Time: O(N * |Labels|)
💾 Space: O(N)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def resolve_position_swap(vote_ab: str, vote_ba: str) -> str:
"""
Resolve pairwise LLM-as-a-Judge verdicts after swapping candidate order (A, B) -> (B, A).
Votes are 'A', 'B', or 'TIE'. Only consistent winners survive; flipped votes become 'TIE'.
"""
if vote_ab == "A" and vote_ba == "B":
return "A"
if vote_ab == "B" and vote_ba == "A":
return "B"
return "TIE"
def cohens_kappa(rater1: list, rater2: list, labels: list = None) -> float:
"""
Compute Cohen's Kappa agreement between LLM judge and human annotators.
"""
r1 = np.asarray(rater1)
r2 = np.asarray(rater2)
n = len(r1)
if n == 0:
return 0.0
if labels is None:
labels = sorted(set(r1.tolist()) | set(r2.tolist()))
p_o =
p_e = 0.0
for lbl in labels:
p1 = float(np.mean(r1 == lbl))
p2 = float(np.mean(r2 == lbl))
p_e += p1 * p2
if abs(1.0 - p_e) < 1e-12:
return 1.0 if abs(p_o - 1.0) < 1e-12 else 0.0
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Beyond position bias, what other systematic biases affect LLM-as-a-Judge and how do you mitigate them?
Q2: When should you choose Pairwise comparison vs. Pointwise 1-5 scoring?
Next: Constrained JSON Decoding via FSM Logit Masking →
💡

Core Mental Anchor / Mnemonic

Swap (A, B) to catch slot bias; subtract chance agreement p_e in Kappa and guard (1 - p_e) near zero.

📐 Mathematical Derivation & Core Formula

κ=po−pe1−pe,pe=∑knk,1⋅nk,2N2\kappa = \frac{p_o - p_e}{1 - p_e}, \quad p_e = \sum_{k} \frac{n_{k,1} \cdot n_{k,2}}{N^2}
### Swap Consistency & Chance-Corrected Agreement
LLM judges frequently exhibit position bias, favoring whichever candidate appears first. Evaluating both (A,B)(A, B) and (B,A)(B, A) reveals ordering bias: if the judge picks the first slot in both runs (vote_ab == 'A' and vote_ba == 'A'), the verdicts contradict and resolve to TIE (or escalate).

Raw agreement pop_o overstates quality under skewed class distributions. Cohen's Kappa adjusts for expected random agreement pe=∑kP1(k)P2(k)p_e = \sum_k P_1(k) P_2(k):
κ=po−pe1−pe\kappa = \frac{p_o - p_e}{1 - p_e}

🔄 Tensor Dimensions & Shape Flow

(vote_ab, vote_ba) -> order-swap consistency check -> verdict; (human, judge) -> marginal p_e -> (p_o - p_e)/(1 - p_e) -> kappa

🛡️ Numerical Stability & Pitfalls Checklist

  • Flag contradictory swapped pairs (e.g., picking first slot twice) as TIE or ambiguous rather than keeping a single pass.
  • Guard the denominator (1 - p_e) against division by zero when both raters collapse onto one label.
  • Pairwise swaps double judge token cost; calibrate on a golden validation subset before rolling out to large runs.

💻 Reference Code Implementation

import numpy as np

def resolve_position_swap(vote_ab: str, vote_ba: str) -> str:
    """
    Resolve pairwise LLM-as-a-Judge verdicts after swapping candidate order (A, B) -> (B, A).
    Votes are 'A', 'B', or 'TIE'. Only consistent winners survive; flipped votes become 'TIE'.
    """
    if vote_ab == "A" and vote_ba == "B":
        return "A"
    if vote_ab == "B" and vote_ba == "A":
        return "B"
    return "TIE"

def cohens_kappa(rater1: list, rater2: list, labels: list = None) -> float:
    """
    Compute Cohen's Kappa agreement between LLM judge and human annotators.
    """
    r1 = np.asarray(rater1)
    r2 = np.asarray(rater2)
    n = len(r1)
    if n == 0:
        return 0.0
    if labels is None:
        labels = sorted(set(r1.tolist()) | set(r2.tolist()))
    p_o = float(np.mean(r1 == r2))
    p_e = 0.0
    for lbl in labels:
        p1 = float(np.mean(r1 == lbl))
        p2 = float(np.mean(r2 == lbl))
        p_e += p1 * p2
    if abs(1.0 - p_e) < 1e-12:
        return 1.0 if abs(p_o - 1.0) < 1e-12 else 0.0
    return float((p_o - p_e) / (1.0 - p_e))

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
# 1. Position-bias detection: always picking first slot ('A' then 'A') must resolve to TIE
assert resolve_position_swap("A", "A") == "TIE", "Position bias must degrade to TIE"
assert resolve_position_swap("A", "B") == "A", "Consistent winner A must pass"
assert resolve_position_swap("B", "A") == "B", "Consistent winner B must pass"
# 2. Cohen's Kappa vs random chance agreement
human = ["A", "A", "B", "B", "A", "B"]
judge = ["A", "A", "B", "A", "A", "B"]
k = cohens_kappa(human, judge, labels=["A", "B"])
# Observed agreement = 5/6 = 0.8333, p_e = (3/6)*(4/6) + (3/6)*(2/6) = 0.5 -> kappa = (5/6 - 0.5)/0.5 = 0.6667
assert np.isclose(k, 2.0 / 3.0, atol=1e-4), f"Unexpected kappa: {k}"
print("✓ LLM-Judge swap & Cohen's Kappa assertion passed")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Beyond position bias, what other systematic biases affect LLM-as-a-Judge and how do you mitigate them?
Verbosity bias (favoring longer answers) and self-preference bias (favoring outputs from the same model family). Mitigate via length-controlled bucketing, rubric checklists penalizing filler, and cross-family judge panels calibrated against human Kappa.
Q2:When should you choose Pairwise comparison vs. Pointwise 1-5 scoring?
Pairwise detects subtle quality shifts between two candidate versions better, while Pointwise scales linearly (O(N)O(N)) and works well for continuous longitudinal monitoring when anchored with explicit rubric criteria.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Moving Average (Sliding Window & Exponential)Back to Part M & All 105Next: Constrained JSON Decoding via FSM Logit Masking →