MRR & GAUC

two industry-grade ranking/recommendation metrics. Mean reciprocal rank $MRR = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{rank_q}$, where $rank_q$ is the position of the first relevant item in the result list of query $q$, contributing 0 when nothing relevant exists; MRR lies in $(0, 1]$. Numerical example: for 3 queries whose first relevant item is at ranks 1, 3 and 5, $MRR = \frac{1}{3}(1 + \frac{1}{3} + \frac{1}{5}) = \frac{1}{3} \times 1.533 \approx 0.511$.

GAUC (Group AUC): compute an AUC per user, then average weighted by each user's sample size, $GAUC = \frac{\sum_u w_u \cdot AUC_u}{\sum_u w_u}$, typically $w_u$ = number of impressions/samples of user $u$. Numerical example: user A has 80 samples with $AUC_A = 0.7$, user B has 20 samples with $AUC_B = 0.6$, so $GAUC = \frac{80 \times 0.7 + 20 \times 0.6}{100} = \frac{56 + 12}{100} = 0.68$, whereas a simple average over users would give $(0.7 + 0.6)/2 = 0.65$.

search/QA/knowledge-retrieval “first hit” evaluation uses MRR; recsys fine-ranking (Alibaba's DIN paper) uses GAUC for per-user ranking quality; interviews often probe the difference, the hand computation, and industrial choice.

MRR cares only about the position of the first relevant result — in search, users barely scroll past the first screen, so extra relevant items and later positions matter little, and MRR focuses attention on “how fast the first hit appears”; global AUC is dominated by heavy active users in recommendation (long-tailed user distribution), so a few power users with good rankings inflate the global AUC and hide poor experience for most users; GAUC evaluates at user granularity and weights each user by sample size, mirroring the “one model per user” personalization view and treating long-tail users more fairly.

Hand-compute MRR: first relevant items at ranks 1, 3 and 5 for 3 queries; what does a query with no relevant result contribute?

MRR vs NDCG: why does MRR only look at the first relevant position, and how does that connect to “search users only look at the first screen”?

Why use GAUC instead of global AUC in recommendation? How does the long-tailed user distribution mislead global AUC?

Hand-compute GAUC: user A with 80 samples at AUC 0.7 and user B with 20 samples at AUC 0.6; show why it differs from the simple average.

Industrial metric selection: which metrics for recall vs fine-ranking stages, and why does the DIN paper use GAUC?