M2-100M2: Classical Machine LearningBias-Variance Tradeoff & Model SelectionMedium
⚡Flashcard Drill➔Mastery:
AIC, BIC, and Adjusted R-squared: Penalized Metrics vs Cross-Validation
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
AIC and BIC penalize model complexity analytically based on likelihood; AIC asymptotically equates to Leave-One-Out CV, while BIC enforces selection consistency.
📌 Key Takeaways
- •AIC: ; optimizes predictive KL-divergence, asymptotically equivalent to LOO-CV
- •BIC: ; penalizes parameters more heavily for , consistent model identification
- •Adjusted : ; degree-of-freedom correction restricted to linear regression
📐 Mathematical Derivations
Mathematical Formulations:
① Akaike Information Criterion (AIC): . Asymptotically estimates the expected Kullback-Leibler (KL) divergence between true data generating distribution and candidate model . Stone (1977) proved that AIC is asymptotically equivalent to Leave-One-Out Cross-Validation (LOO-CV).
② Bayesian Information Criterion (BIC): . Derived from Laplace approximation of the marginal log-likelihood . BIC is consistent: as , the probability of selecting the true model converges to 1 (if true model is in the candidate set).
③ Adjusted $R^2$: . Penalizes adding non-significant predictors but does not apply across non-nested or non-linear models.
① Akaike Information Criterion (AIC): . Asymptotically estimates the expected Kullback-Leibler (KL) divergence between true data generating distribution and candidate model . Stone (1977) proved that AIC is asymptotically equivalent to Leave-One-Out Cross-Validation (LOO-CV).
② Bayesian Information Criterion (BIC): . Derived from Laplace approximation of the marginal log-likelihood . BIC is consistent: as , the probability of selecting the true model converges to 1 (if true model is in the candidate set).
③ Adjusted $R^2$: . Penalizes adding non-significant predictors but does not apply across non-nested or non-linear models.
🏭 Production Trade-offs
Efficiency vs Assumptions: AIC and BIC require only a single training fit on the full dataset, making them computationally instant compared to -fold cross-validation. However, they rely strictly on parametric likelihood assumptions. -fold CV is non-parametric, robust to model mis-specification, and directly measures test error.
⚠️ Common Interview Pitfalls
- ✕Comparing AIC or BIC scores across models fit using different sample sizes or different response variable transformations
- ✕Assuming lower AIC implies the simpler model is true; AIC prioritizes predictive accuracy and tends to overfit compared to BIC
🎯 Interviewer Follow-up Inquiries
- ?Why is AIC asymptotically equivalent to Leave-One-Out Cross-Validation (LOO-CV)?
- ?Under what conditions will BIC consistently recover the true data-generating model while AIC overfits?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚