M2-018M2: Classical Machine LearningBias-Variance Tradeoff & Model SelectionMedium
⚡Flashcard Drill➔Mastery:

Compare K-Fold Cross-Validation, Leave-One-Out (LOO), and Holdout Splits in Bias, Variance, and Compute

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-019 →
📐 Mathematical Definition
CV: err^=1K∑kerrk\text{CV}:\ \hat{\mathrm{err}}=\frac1K\sum_k\mathrm{err}_k
⚡ One-Sentence Core Answer
Holdout is fast but high-variance; LOO is nearly unbiased but computationally prohibitive with high estimator variance; K-Fold (K=5K=5 or 1010) strikes the optimal empirical compromise.

📌 Key Takeaways

  • •
    Holdout (e.g. 80/20): Single split; fastest (O(1)O(1) training), but estimate is sensitive to the specific split and wastes 20% of data.
  • •
    Leave-One-Out (LOO, K=NK=N): Trains NN separate models on N−1N-1 samples; essentially unbiased, but O(N)O(N) training cost.
  • •
    K-Fold (K=5K=5 or 1010): Standard industrial default; balances low bias with moderate variance and manageable O(K)O(K) computation.

📐 Mathematical Derivations

Bias and Variance tradeoffs across cross-validation methods: (1) Bias: In KK-fold CV, each fold trains on K−1KN\frac{K-1}{K}N samples. Because learning curves are concave, training on less than NN samples overestimates true generalization error (pessimistic bias). LOO trains on N−1N-1 samples, exhibiting virtually zero bias (E[ErrLOO]≈ErrNE[\text{Err}_{\text{LOO}}] \approx \text{Err}_N). (2) Variance: In LOO, the NN training sets are virtually identical (sharing N−2N-2 samples). Consequently, the NN validation error estimates are strongly positively correlated. Since Var(1N∑Zi)=σ2N+N−1NCov(Zi,Zj)\text{Var}\left(\frac{1}{N}\sum Z_i\right) = \frac{\sigma^2}{N} + \frac{N-1}{N}\text{Cov}(Z_i, Z_j), high positive covariance causes the variance of the LOO estimator to be surprisingly large! K=10K=10 folds have less overlapping training sets, mitigating covariance.

🏭 Production Trade-offs

Specialized analytical shortcuts: For linear regression, the Sherman-Morrison-Woodbury theorem computes exact LOO error in a single training pass: yi−y^(−i)=yi−y^i1−hiiy_i - \hat{y}_{(-i)} = \frac{y_i - \hat{y}_i}{1 - h_{ii}}, where hiih_{ii} are the diagonal leverage scores of the hat matrix HH. This is known as Generalized Cross-Validation (GCV).
⚠️ Common Interview Pitfalls
  • ✕
    Failing to use Stratified K-Fold for imbalanced classification (strata must preserve class proportions).
  • ✕
    Using standard K-Fold on time-series data (causes severe data leakage from the future; TimeSeriesSplit / rolling-window must be used).
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Generalized Cross-Validation (GCV) evaluate LOO error in a single matrix inversion for OLS?
  • ?
    Why does Stratified K-Fold provide lower variance for imbalanced datasets?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-017: Detail How to Diagnose High Bias vs. High Variance Using Learning Curves and Remediation Strategies📋 Back to BankNext →M2-019: Explain Nested Cross-Validation and How It Resolves Hyperparameter Selection Optimism Bias