M2-018M2: Classical Machine LearningBias-Variance Tradeoff & Model SelectionMedium
⚡Flashcard Drill➔Mastery:
Compare K-Fold Cross-Validation, Leave-One-Out (LOO), and Holdout Splits in Bias, Variance, and Compute
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Holdout is fast but high-variance; LOO is nearly unbiased but computationally prohibitive with high estimator variance; K-Fold ( or ) strikes the optimal empirical compromise.
📌 Key Takeaways
- •Holdout (e.g. 80/20): Single split; fastest ( training), but estimate is sensitive to the specific split and wastes 20% of data.
- •Leave-One-Out (LOO, ): Trains separate models on samples; essentially unbiased, but training cost.
- •K-Fold ( or ): Standard industrial default; balances low bias with moderate variance and manageable computation.
📐 Mathematical Derivations
Bias and Variance tradeoffs across cross-validation methods: (1) Bias: In -fold CV, each fold trains on samples. Because learning curves are concave, training on less than samples overestimates true generalization error (pessimistic bias). LOO trains on samples, exhibiting virtually zero bias (). (2) Variance: In LOO, the training sets are virtually identical (sharing samples). Consequently, the validation error estimates are strongly positively correlated. Since , high positive covariance causes the variance of the LOO estimator to be surprisingly large! folds have less overlapping training sets, mitigating covariance.
🏭 Production Trade-offs
Specialized analytical shortcuts: For linear regression, the Sherman-Morrison-Woodbury theorem computes exact LOO error in a single training pass: , where are the diagonal leverage scores of the hat matrix . This is known as Generalized Cross-Validation (GCV).
⚠️ Common Interview Pitfalls
- ✕Failing to use Stratified K-Fold for imbalanced classification (strata must preserve class proportions).
- ✕Using standard K-Fold on time-series data (causes severe data leakage from the future; TimeSeriesSplit / rolling-window must be used).
🎯 Interviewer Follow-up Inquiries
- ?How does Generalized Cross-Validation (GCV) evaluate LOO error in a single matrix inversion for OLS?
- ?Why does Stratified K-Fold provide lower variance for imbalanced datasets?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚