⚡ LLMsID: benchmarks

Benchmarks MMLU/GSM8K

Q: What capability does each of MMLU/GSM8K/HumanEval/IFEval measure, and why 5-shot rather than zero-shot?

【Benchmarks MMLU/GSM8K 核心考点 1】What capability does each of MMLU/GSM8K/HumanEval/IFEval measure, and why 5-shot rather than zero-shot?。技术解析：Definition: four representative benchmarks anchor different capability axes: (1) MMLU — 57-subject multiple choice, 5-shot CoT, world knowledge and reasoning breadth; (2) GSM8K — 8,000 grade-school math word problems requiring stepwise reasoning (largest CoT gains), arithmetic an... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/benchmarks

Q: Derive pass@k: with c passing out of n samples, why does the binomial formula give the correct estimate?

【Benchmarks MMLU/GSM8K 核心考点 2】Derive pass@k: with c passing out of n samples, why does the binomial formula give the correct estimate?。技术解析：Definition: four representative benchmarks anchor different capability axes: (1) MMLU — 57-subject multiple choice, 5-shot CoT, world knowledge and reasoning breadth; (2) GSM8K — 8,000 grade-school math word problems requiring stepwise reasoning (largest CoT gains), arithmetic an... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/benchmarks

Q: What are the signs and causes of benchmark saturation, and how do you respond (harder variants, dynamic evals, leakage control)?

【Benchmarks MMLU/GSM8K 核心考点 3】What are the signs and causes of benchmark saturation, and how do you respond (harder variants, dynamic evals, leakage control)?。技术解析：Definition: four representative benchmarks anchor different capability axes: (1) MMLU — 57-subject multiple choice, 5-shot CoT, world knowledge and reasoning breadth; (2) GSM8K — 8,000 grade-school math word problems requiring stepwise reasoning (largest CoT gains), arithmetic an... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/benchmarks

Q: How do you detect and avoid data contamination — dedup, public timelines, held-out sets?

【Benchmarks MMLU/GSM8K 核心考点 4】How do you detect and avoid data contamination — dedup, public timelines, held-out sets?。技术解析：Definition: four representative benchmarks anchor different capability axes: (1) MMLU — 57-subject multiple choice, 5-shot CoT, world knowledge and reasoning breadth; (2) GSM8K — 8,000 grade-school math word problems requiring stepwise reasoning (largest CoT gains), arithmetic an... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/benchmarks

Q: GSM8K gains hugely from CoT yet trivial formatting changes help too — what does that imply, and how do you control prompt sensitivity in eval?

【Benchmarks MMLU/GSM8K 核心考点 5】GSM8K gains hugely from CoT yet trivial formatting changes help too — what does that imply, and how do you control prompt sensitivity in eval?。技术解析：Definition: four representative benchmarks anchor different capability axes: (1) MMLU — 57-subject multiple choice, 5-shot CoT, world knowledge and reasoning breadth; (2) GSM8K — 8,000 grade-school math word problems requiring stepwise reasoning (largest CoT gains), arithmetic an... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/benchmarks

评测基准 MMLU/GSM8K

🎯Core Definition

four representative benchmarks anchor different capability axes: (1) MMLU — 57-subject multiple choice, 5-shot CoT, world knowledge and reasoning breadth; (2) GSM8K — 8,000 grade-school math word problems requiring stepwise reasoning (largest CoT gains), arithmetic and multi-step reasoning; (3) HumanEval — 164 hand-written Python function-completion tasks, scored with pass@k:

\text{pass}@k = 1 - \binom{n-c}{k} / \binom{n}{k}

(

c

correct out of

n

samples); (4) IFEval — verifiable instruction following (format/length/keyword constraints), testing obedience rather than answer correctness.

💡Use Cases

model selection and version comparison (same 5-shot protocol); paper baseline reporting; regression testing after SFT/RL; interview staple — what each benchmark measures and how to judge who wins once scores saturate.

⚡Key Problems Solved

one aggregate score can't cover the capability space; the four anchor knowledge breadth, math reasoning, code generation, and instruction following. But closed-book MCQ scores are near saturation (some leaderboards >95%) and suffer training-data contamination — public stems absorbed into pretraining data inflate scores, pushing the field to harder variants (MMLU-Pro, GSM-Plus) and leakage-controlled evaluation.

🎯5 High-Frequency Exam Points

What capability does each of MMLU/GSM8K/HumanEval/IFEval measure, and why 5-shot rather than zero-shot?

Derive pass@k: with c passing out of n samples, why does the binomial formula give the correct estimate?

What are the signs and causes of benchmark saturation, and how do you respond (harder variants, dynamic evals, leakage control)?

How do you detect and avoid data contamination — dedup, public timelines, held-out sets?

GSM8K gains hugely from CoT yet trivial formatting changes help too — what does that imply, and how do you control prompt sensitivity in eval?

📖 In-depth Guide:📄 llm-foundations-and-sota →

Updated 2026-08-12

🎯

Test Your Knowledge: Practice Questions for "Benchmarks MMLU/GSM8K"

Single choice pitfall questions with instant feedback and mistake tracking.

🚀 Start Card Practice ➔

← Previous CardAgent & Tool Calling Next Card →Auto Metrics vs LLM-as-Judge

🔗 More LLMs Knowledge Cards

Alignment Tax & Preference Data Scaled Dot-Product Attention Attention Variants MHA/MQA/GQA Tokenization: BPE / WordPiece