M2-089M2: Classical Machine LearningEvaluation Metrics & Hyperparameter TuningHard
⚡Flashcard Drill➔Mastery:
Handling Randomness and Seed Variance in Hyperparameter Tuning
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Evaluate configurations across multiple random seeds, report mean and confidence intervals, and apply paired statistical tests to distinguish genuine improvements from seed noise.
📌 Key Takeaways
- •Sources of randomness: weight initialization, mini-batch shuffling, data splits, non-deterministic GPU operations
- •Seed variance risk: '0.5% gain' may simply reflect an outlier favorable random seed
- •Paired comparison: evaluate candidate models using identical random seeds and data folds to cancel shared noise
📐 Mathematical Derivations
Statistical Formulation: Let performance metric of model on seed be , where . If seed variance is and the true improvement , a single-seed trial has a high probability of falsely declaring model inferior.
Resolution via Averaging: Evaluating across independent seeds reduces standard error of the mean to . To detect effect size with statistical power at , required seeds: .
Paired $t$-test: Compute seed-wise differences using identical seeds, testing via .
Resolution via Averaging: Evaluating across independent seeds reduces standard error of the mean to . To detect effect size with statistical power at , required seeds: .
Paired $t$-test: Compute seed-wise differences using identical seeds, testing via .
🏭 Production Trade-offs
Standardized best practices: ① Never report the single best seed (which represents extreme selection bias / lucky draw). Always report over at least 3–5 seeds. ② Budget allocation: If evaluation noise is large, reduce the number of tested hyperparameter configurations and allocate compute to multi-seed verification. ③ Reproducibility: Set deterministic flags (e.g.,
torch.use_deterministic_algorithms(True)) during regression tests, though note small training speed penalties.⚠️ Common Interview Pitfalls
- ✕Picking the single best random seed from 20 runs and reporting it as the model's actual performance benchmark
- ✕Declaring an architectural improvement based on a 0.2% metric bump when single-seed standard deviation is 0.6%
🎯 Interviewer Follow-up Inquiries
- ?Why is a paired -test statistically more powerful than an two-sample independent -test when comparing models across seeds?
- ?How does selection bias affect reported benchmarks when picking the maximum validation score across 100 random seeds?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚