M2-100M2: Classical Machine LearningBias-Variance Tradeoff & Model SelectionMedium
⚡Flashcard Drill➔Mastery:

AIC, BIC, and Adjusted R-squared: Penalized Metrics vs Cross-Validation

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-101 →
📐 Mathematical Definition
AIC=2k−2log⁡L^,BIC=klog⁡n−2log⁡L^\text{AIC}=2k-2\log\hat L,\qquad \text{BIC}=k\log n-2\log\hat L
⚡ One-Sentence Core Answer
AIC and BIC penalize model complexity analytically based on likelihood; AIC asymptotically equates to Leave-One-Out CV, while BIC enforces selection consistency.

📌 Key Takeaways

  • •
    AIC: 2k−2ln⁡L^2k - 2\ln \hat{L}; optimizes predictive KL-divergence, asymptotically equivalent to LOO-CV
  • •
    BIC: kln⁡n−2ln⁡L^k \ln n - 2\ln \hat{L}; penalizes parameters more heavily for n>7n > 7, consistent model identification
  • •
    Adjusted R2R^2: 1−(1−R2)(n−1)n−k−11 - \frac{(1-R^2)(n-1)}{n-k-1}; degree-of-freedom correction restricted to linear regression

📐 Mathematical Derivations

Mathematical Formulations:
① Akaike Information Criterion (AIC): AIC=2k−2ln⁡L^\text{AIC} = 2k - 2\ln \hat{L}. Asymptotically estimates the expected Kullback-Leibler (KL) divergence between true data generating distribution g(x)g(x) and candidate model f(x∣θ^)f(x|\hat{\theta}). Stone (1977) proved that AIC is asymptotically equivalent to Leave-One-Out Cross-Validation (LOO-CV).
② Bayesian Information Criterion (BIC): BIC=kln⁡n−2ln⁡L^\text{BIC} = k \ln n - 2\ln \hat{L}. Derived from Laplace approximation of the marginal log-likelihood ln⁡P(D∣M)\ln P(D|M). BIC is consistent: as n→∞n \to \infty, the probability of selecting the true model converges to 1 (if true model is in the candidate set).
③ Adjusted $R^2$: Rˉ2=1−SSres/(n−k−1)SStot/(n−1)\bar{R}^2 = 1 - \frac{\text{SS}_{\text{res}} / (n - k - 1)}{\text{SS}_{\text{tot}} / (n - 1)}. Penalizes adding non-significant predictors but does not apply across non-nested or non-linear models.

🏭 Production Trade-offs

Efficiency vs Assumptions: AIC and BIC require only a single training fit on the full dataset, making them computationally instant compared to KK-fold cross-validation. However, they rely strictly on parametric likelihood assumptions. KK-fold CV is non-parametric, robust to model mis-specification, and directly measures test error.
⚠️ Common Interview Pitfalls
  • ✕
    Comparing AIC or BIC scores across models fit using different sample sizes nn or different response variable transformations
  • ✕
    Assuming lower AIC implies the simpler model is true; AIC prioritizes predictive accuracy and tends to overfit compared to BIC
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is AIC asymptotically equivalent to Leave-One-Out Cross-Validation (LOO-CV)?
  • ?
    Under what conditions will BIC consistently recover the true data-generating model while AIC overfits?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-099: Stochastic Depth (DropPath) vs Standard Dropout: Mechanisms and Differences📋 Back to BankNext →M2-101: Learning Curves vs Validation Curves: Formulations and Diagnostic Workflows