⚡Flashcard Drill➔
L0S5 · Eval & LLMOpsVerified: 2026-11-17

LLM-as-a-Judge Bias Mitigation & Cohen's Kappa Calibration Primitive

Evaluation calibration primitive countering position bias, verbosity inflation, and self-enhancement bias in LLM judges through A/B position swapping, integer rubric anchoring, and human Cohen's kappa verification.

Best For

Evaluating open-ended synthesis, customer-facing explanations, and model-upgrade comparisons where regex or exact-match metrics fall short.

Human-AI Boundary

Every rubric edit must re-pass Cohen's kappa alignment against domain-expert blind labels before gating production releases.

Anti-Patterns to Avoid

  • Asking an LLM judge to output vague 1-10 floating scores without observable pass/fail rubric criteria or position-swapping checks
  • Using the same model family being promoted (e.g. Candidate Model X) as the uncalibrated sole judge without checking human agreement

Latency/Cost Profile & Eval Metrics

Offline CI / nightly batch; 2x judge calls when running symmetric A/B swap verification · Judge frontier-model tokens x 2 (swapped order) on ambiguous pairwise slices

  • Judge-Human Alignment (Cohen's Kappa): >= 0.75 on stratified 100-sample human-labeled calibration slice (Re-calibrate whenever prompt rubric or judge model checkpoint changes)
  • Position-Swap Consistency Rate: >= 90% consistent winner after swapping (A, B) -> (B, A) (Treat order-flipping judgments as explicit ties rather than trusting A-first bias)

💰 Business ROI Model & Evidence Level

verified_benchmark

📊 Benchmark & Alternative Comparison

  • Binary Pass/Fail Criterion Judges: higher inter-annotator consistency and actionable error taxonomies than fuzzy 1-5 Likert scales.
  • Symmetric Pairwise with Position Swap: exposes small quality regressions cleanly when comparing two system versions head-to-head.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔

Bidirectional Taxonomy Links

[L1] Stage 1 · Shadow-Mode AI Baseline Validation Pattern Before Live Automation[L1] Stage 2 · Semantic-Layer Grounded Text-to-SQL & Read-Only Sandbox Repair Pattern[L1] Stage 5 · Canary Prompt & Model-Upgrade Shadow Traffic A/B Gate Pattern[L1] Stage 5 · Production Trace-to-Golden-Dataset Failure Flywheel Pattern[L2] L2 System Design · Sub-15ms Global Payment Fraud Real-Time ML Architecture[HubA] Company Intelligence Hub · Google DeepMind, Gemini & Cloud AI Loop[HubA] Company Intelligence Hub · Meta AI/MLE, Ranking & Infra Loop[HubB] Role Lens · Applied Scientist (Research-to-Production & Post-Training)[L0 Card] Judge Position Bias Mitigation[L0 Card] Verbosity & Self-Enhancement Bias[L0 Card] Cohen's Kappa Agreement[L0 Card] Pairwise A/B Comparison[L0 Card] Pointwise Evaluation Paradigm

Internal Tools & Primary Sources