M2-103M2: Classical Machine LearningBias-Variance Tradeoff & Model SelectionMedium
⚡Flashcard Drill➔Mastery:

When Does Adding More Data Fail to Improve Model Performance?

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-104 →
📐 Mathematical Definition
E[err]=Bias2+Var+σ2\mathbb E[\text{err}]=\mathrm{Bias}^2+\mathrm{Var}+\sigma^2
⚡ One-Sentence Core Answer
Adding data fails when performance is bottlenecked by high model bias, irreducible Bayes noise, distribution shift, or systematic label corruption.

📌 Key Takeaways

  • •
    High model bias: hypothesis space H\mathcal{H} lacks expressive capacity to represent the true function
  • •
    Irreducible noise: Bayes error rate σ2=E[Var(Y∣X)]\sigma^2 = \mathbb{E}[\text{Var}(Y|X)] forms a theoretical lower bound on loss
  • •
    Distribution mismatch: adding training data drawn from a different distribution Ptrain≠PtestP_{\text{train}} \ne P_{\text{test}}

📐 Mathematical Derivations

Decomposition Analysis: Generalization error decomposes as: Risk(f)=Bias2(H)+Variance(n)+σirreducible2\text{Risk}(f) = \text{Bias}^2(\mathcal{H}) + \text{Variance}(n) + \sigma^2_{\text{irreducible}}.
- As n→∞n \to \infty, Variance(n)→0\text{Variance}(n) \to 0. However, Bias2\text{Bias}^2 and σirreducible2\sigma^2_{\text{irreducible}} remain strictly invariant with respect to nn.
- If Bias2≫Variance\text{Bias}^2 \gg \text{Variance}, the learning curve has already reached its asymptotic plateau. Adding 10×10\times more samples will only reduce the already negligible variance term, leaving test error effectively unchanged.
- Label Noise Ceiling: If random label flip probability is η\eta, minimum achievable misclassification error is η\eta, which no volume of training examples can overcome.

🏭 Production Trade-offs

Diagnostic checklist: When data scaling yields no lift: ① Check learning curve slope; ② Evaluate training error (if training error is high, increase model complexity); ③ Audit label quality and data agreement between annotators; ④ Test for covariate shift between train and serving distributions.
⚠️ Common Interview Pitfalls
  • ✕
    Investing large engineering budgets in gathering more data for an underfitting model instead of upgrading model capacity
  • ✕
    Collecting abundant web-scraped data that has significant domain divergence from target production traffic
🎯 Interviewer Follow-up Inquiries
  • ?
    How do you mathematically determine whether an error gap is driven by Bayes error or model bias?
  • ?
    What strategies can filter out systematic label noise before retraining on massive datasets?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-102: How Model Ensembling Impacts Bias and Variance📋 Back to BankNext →M2-104: The 'No Free Lunch' Theorem and Inductive Bias Explained