M2-075M2: Classical Machine LearningClass Imbalance LearningHard
⚡Flashcard Drill➔Mastery:

Cost-Sensitive Learning: Implementation and Relationship with Re-sampling

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-076 →
📐 Mathematical Definition
L=−∑iwyilog⁡pyi\mathcal L=-\sum_i w_{y_i}\log p_{y_i}
⚡ One-Sentence Core Answer
Incorporate class weights into the loss function; it is asymptotically equivalent to re-sampling but avoids data duplication and stabilizes empirical training variance.

📌 Key Takeaways

  • •
    Loss weighting: L=−∑iwyiℓ(yi,y^i)\mathcal{L} = - \sum_i w_{y_i} \ell(y_i, \hat{y}_i) with inverse class frequencies
  • •
    Equivalence with re-sampling: weighting an instance by rr contributes the same expected gradient as sampling it rr times
  • •
    Key divergence: re-sampling alters the empirical data distribution, affecting calibration and batch norm statistics

📐 Mathematical Derivations

Mathematical Connection: Expected risk under importance sampling: Ex∼P′[ℓ(x,y)]=Ex∼P[P′(x,y)P(x,y)ℓ(x,y)]\mathbb{E}_{x \sim P'}[\ell(x, y)] = \mathbb{E}_{x \sim P}\left[\frac{P'(x, y)}{P(x, y)} \ell(x, y)\right]. Oversampling the minority class by factor rr scales its empirical frequency from PP to P′P', which corresponds to multiplying its loss term by w=rw = r.
Divergence Points: ① Variance: Oversampling draws duplicate samples with replacement, introducing discrete covariance across mini-batch gradients; class weighting applies deterministic scaling without inflating batch sample count. ② Calibration: Re-sampling shifts the model's internal prior towards 50/50, requiring post-hoc log-odds correction: logit(p)=logit(presampled)−log⁡(r)\text{logit}(p) = \text{logit}(p_{\text{resampled}}) - \log(r). Class weighting maintains raw feature density.

🏭 Production Trade-offs

Implementation preference: In modern production pipelines, Class Weights or Focal Loss are preferred over oversampling due to zero memory overhead, reproducible deterministic batches, and absence of synthetic sample artifacts. Reserve undersampling for situations where the majority class is overwhelmingly large (>108>10^8 rows) and compute budget is the primary bottleneck.
⚠️ Common Interview Pitfalls
  • ✕
    Treating class-weighted model outputs as raw calibrated probabilities without performing Bayesian prior correction
  • ✕
    Believing oversampling provides new information, whereas it simply repeats existing samples and increases overfitting risk
🎯 Interviewer Follow-up Inquiries
  • ?
    How do you mathematically correct the output probabilities of a model trained on an undersampled dataset?
  • ?
    Under what conditions does Focal Loss provide superior gradients compared to simple inverse class weighting?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-074: Classification Threshold Selection Strategies Across Diverse Business Objectives📋 Back to BankNext →M2-076: Three Missing Data Mechanisms (MCAR, MAR, MNAR) and Treatment Strategies