L0S5 · Eval & LLMOpsVerified: 2026-11-17
LLM-as-a-Judge Bias Mitigation & Cohen's Kappa Calibration Primitive
Evaluation calibration primitive countering position bias, verbosity inflation, and self-enhancement bias in LLM judges through A/B position swapping, integer rubric anchoring, and human Cohen's kappa verification.
Best For
Evaluating open-ended synthesis, customer-facing explanations, and model-upgrade comparisons where regex or exact-match metrics fall short.
Human-AI Boundary
Every rubric edit must re-pass Cohen's kappa alignment against domain-expert blind labels before gating production releases.
Anti-Patterns to Avoid
- Asking an LLM judge to output vague 1-10 floating scores without observable pass/fail rubric criteria or position-swapping checks
- Using the same model family being promoted (e.g. Candidate Model X) as the uncalibrated sole judge without checking human agreement
Latency/Cost Profile & Eval Metrics
Offline CI / nightly batch; 2x judge calls when running symmetric A/B swap verification · Judge frontier-model tokens x 2 (swapped order) on ambiguous pairwise slices
- Judge-Human Alignment (Cohen's Kappa): >= 0.75 on stratified 100-sample human-labeled calibration slice (Re-calibrate whenever prompt rubric or judge model checkpoint changes)
- Position-Swap Consistency Rate: >= 90% consistent winner after swapping (A, B) -> (B, A) (Treat order-flipping judgments as explicit ties rather than trusting A-first bias)
💰 Business ROI Model & Evidence Level
verified_benchmark📊 Benchmark & Alternative Comparison
- Binary Pass/Fail Criterion Judges: higher inter-annotator consistency and actionable error taxonomies than fuzzy 1-5 Likert scales.
- Symmetric Pairwise with Position Swap: exposes small quality regressions cleanly when comparing two system versions head-to-head.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
Bidirectional Taxonomy Links
[L1] Stage 1 · Shadow-Mode AI Baseline Validation Pattern Before Live Automation[L1] Stage 2 · Semantic-Layer Grounded Text-to-SQL & Read-Only Sandbox Repair Pattern[L1] Stage 5 · Canary Prompt & Model-Upgrade Shadow Traffic A/B Gate Pattern[L1] Stage 5 · Production Trace-to-Golden-Dataset Failure Flywheel Pattern[L2] L2 System Design · Sub-15ms Global Payment Fraud Real-Time ML Architecture[HubA] Company Intelligence Hub · Google DeepMind, Gemini & Cloud AI Loop[HubA] Company Intelligence Hub · Meta AI/MLE, Ranking & Infra Loop[HubB] Role Lens · Applied Scientist (Research-to-Production & Post-Training)[L0 Card] Judge Position Bias Mitigation[L0 Card] Verbosity & Self-Enhancement Bias[L0 Card] Cohen's Kappa Agreement[L0 Card] Pairwise A/B Comparison[L0 Card] Pointwise Evaluation Paradigm
Internal Tools & Primary Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al.) (paper, 2026-11-17)
- Hamel Husain — Your AI Product Needs Evals (Judge Calibration) (engineering_blog, 2026-11-17)