K-Fold Cross-Validation

K-Fold cross-validation splits the dataset into K equal folds, trains on K-1 and validates on the remaining one in rotation; the error estimate is the mean over folds $\hat{E} = \frac{1}{K}\sum_{k=1}^{K} E_k$ where $E_k$ is the validation error of fold k. Key variants: Stratified K-Fold (each fold keeps the global class proportion, mandatory for classification), Group K-Fold (samples of the same group — same user or experiment — never span folds, preventing group-correlation leakage), and leave-one-out (LOOCV, K=N). Bias-variance tradeoff: larger K means more training data and lower bias, but more overlap between training sets and higher variance; K=5 or 10 is the usual compromise.

estimating generalization with limited data; comparing configurations for hyperparameter tuning, model selection and feature selection (pick the best CV score); interviews ask about choosing K, stratified vs group, and how comparing CV error with training error reveals overfitting.

a single holdout split depends on luck and has high variance (the same model can differ by several points across splits); K-Fold validates every sample exactly once, giving a more stable and trustworthy estimate; compared with LOOCV ($O(N)$ trainings, expensive and high variance), K-Fold trades bias against variance and is the default standard for ML evaluation and selection.

Write the K-Fold error estimate $\hat{E} = \frac{1}{K}\sum_{k=1}^{K} E_k$ and explain why K=5 or 10 beats LOOCV (bias-variance angle)?

Stratified vs plain K-Fold? Why does ignoring stratification hurt classification? When must you use Group K-Fold?

Why does larger K reduce bias but increase variance? How does training-set overlap between folds affect variance?

CV and leakage: why must scaling, imputation and feature selection be done inside each fold?

What does a training error far below the CV error tell you? How does CV error relate to test error?