Elo Rating System

The Elo Rating System (popularized in AI evaluation by LMSYS Chatbot Arena) is an adaptive ranking framework that computes global capability scores for competing LLMs via aggregated crowdsourced blind pairwise battles; given current ratings $R_A, R_B$, model A's expected win probability is $E_A = \frac{1}{1 + 10^{(R_B - R_A)/400}}$; following a battle with outcome $S_A$ (Win=1, Tie=0.5, Loss=0), ratings update dynamically via $R_A^{\prime} = R_A + K \cdot (S_A - E_A)$, where $K$ controls update magnitude.

Universal frontier LLM leaderboards (LMSYS Chatbot Arena), continuous checkpoint benchmarking, and open-ended preference tracking.

Static multiple-choice benchmarks suffer severe dataset contamination and benchmark hacking; live blind crowd-sourced Elo evaluation dynamically captures genuine general reasoning and human preference under zero data contamination.

Derive the Elo expected win-rate and rating update formulas, and analyze the score surge when an underdog upsets a top model?

Compare Bradley-Terry Maximum Likelihood Estimation (MLE) vs online incremental Elo updates in rating convergence?

How does non-parametric Bootstrapping compute robust 95% Confidence Intervals across model Elo ratings?

How to structure category-specific Elo sub-arenas (e.g., Coding Arena, Hard Prompts, Multi-turn)?

How does the Glicko-2 algorithm utilize Rating Deviation (RD) to gracefully handle cold-start variance in new models?