Benchmarks MMLU/GSM8K

four representative benchmarks anchor different capability axes: (1) MMLU — 57-subject multiple choice, 5-shot CoT, world knowledge and reasoning breadth; (2) GSM8K — 8,000 grade-school math word problems requiring stepwise reasoning (largest CoT gains), arithmetic and multi-step reasoning; (3) HumanEval — 164 hand-written Python function-completion tasks, scored with pass@k: $\text{pass}@k = 1 - \binom{n-c}{k} / \binom{n}{k}$ ($c$ correct out of $n$ samples); (4) IFEval — verifiable instruction following (format/length/keyword constraints), testing obedience rather than answer correctness.

model selection and version comparison (same 5-shot protocol); paper baseline reporting; regression testing after SFT/RL; interview staple — what each benchmark measures and how to judge who wins once scores saturate.

one aggregate score can't cover the capability space; the four anchor knowledge breadth, math reasoning, code generation, and instruction following. But closed-book MCQ scores are near saturation (some leaderboards >95%) and suffer training-data contamination — public stems absorbed into pretraining data inflate scores, pushing the field to harder variants (MMLU-Pro, GSM-Plus) and leakage-controlled evaluation.

What capability does each of MMLU/GSM8K/HumanEval/IFEval measure, and why 5-shot rather than zero-shot?

Derive pass@k: with c passing out of n samples, why does the binomial formula give the correct estimate?

What are the signs and causes of benchmark saturation, and how do you respond (harder variants, dynamic evals, leakage control)?

How do you detect and avoid data contamination — dedup, public timelines, held-out sets?

GSM8K gains hugely from CoT yet trivial formatting changes help too — what does that imply, and how do you control prompt sensitivity in eval?