M2-089M2: Classical Machine LearningEvaluation Metrics & Hyperparameter TuningHard
⚡Flashcard Drill➔Mastery:

Handling Randomness and Seed Variance in Hyperparameter Tuning

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-090 →
📐 Mathematical Definition
σ^seed2=1S∑s(perfs−perfˉ)2\hat\sigma^2_{seed}=\frac1S\sum_s(\mathrm{perf}_s-\bar{\mathrm{perf}})^2
⚡ One-Sentence Core Answer
Evaluate configurations across multiple random seeds, report mean and confidence intervals, and apply paired statistical tests to distinguish genuine improvements from seed noise.

📌 Key Takeaways

  • •
    Sources of randomness: weight initialization, mini-batch shuffling, data splits, non-deterministic GPU operations
  • •
    Seed variance risk: '0.5% gain' may simply reflect an outlier favorable random seed
  • •
    Paired comparison: evaluate candidate models using identical random seeds and data folds to cancel shared noise

📐 Mathematical Derivations

Statistical Formulation: Let performance metric of model AA on seed ss be XA,s=μA+ϵsX_{A, s} = \mu_A + \epsilon_s, where ϵs∼N(0,σ2)\epsilon_s \sim \mathcal{N}(0, \sigma^2). If seed variance σ\sigma is 1.0%1.0\% and the true improvement Δ=μA−μB=0.5%\Delta = \mu_A - \mu_B = 0.5\%, a single-seed trial has a high probability P(XA,s<XB,s)=Φ(−Δ2σ)=Φ(−0.35)≈36.3%P(X_{A, s} < X_{B, s}) = \Phi\left(-\frac{\Delta}{\sqrt{2}\sigma}\right) = \Phi(-0.35) \approx 36.3\% of falsely declaring model AA inferior.
Resolution via Averaging: Evaluating across SS independent seeds reduces standard error of the mean to σ/S\sigma / \sqrt{S}. To detect effect size Δ\Delta with statistical power 1−β=0.801 - \beta = 0.80 at α=0.05\alpha = 0.05, required seeds: S≥2(zα/2+zβΔ/σ)2S \ge 2 \left( \frac{z_{\alpha/2} + z_\beta}{\Delta / \sigma} \right)^2.
Paired $t$-test: Compute seed-wise differences Ds=XA,s−XB,sD_s = X_{A, s} - X_{B, s} using identical seeds, testing H0:μD=0H_0: \mu_D = 0 via t=DˉsD/St = \frac{\bar{D}}{s_D / \sqrt{S}}.

🏭 Production Trade-offs

Standardized best practices: ① Never report the single best seed (which represents extreme selection bias / lucky draw). Always report Mean±Std\text{Mean} \pm \text{Std} over at least 3–5 seeds. ② Budget allocation: If evaluation noise is large, reduce the number of tested hyperparameter configurations and allocate compute to multi-seed verification. ③ Reproducibility: Set deterministic flags (e.g., torch.use_deterministic_algorithms(True)) during regression tests, though note small training speed penalties.
⚠️ Common Interview Pitfalls
  • ✕
    Picking the single best random seed from 20 runs and reporting it as the model's actual performance benchmark
  • ✕
    Declaring an architectural improvement based on a 0.2% metric bump when single-seed standard deviation is 0.6%
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is a paired tt-test statistically more powerful than an two-sample independent tt-test when comparing models across seeds?
  • ?
    How does selection bias affect reported benchmarks when picking the maximum validation score across 100 random seeds?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-088: Hyperparameter Importance Hierarchy and Efficient Tuning Strategies📋 Back to BankNext →M2-090: Why Offline and Online Evaluations Diverge: Causes and Mitigation Strategies