M2-074M2: Classical Machine LearningClass Imbalance LearningMedium
⚡Flashcard Drill➔Mastery:
Classification Threshold Selection Strategies Across Diverse Business Objectives
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Select thresholds by minimizing expected cost via cost matrices or meeting business constraints (e.g., recall floor) using a calibrated validation set.
📌 Key Takeaways
- •Cost minimization: select
- •Constraint optimization: maximize Precision subject to (or vice-versa)
- •Probability calibration: thresholds have stable business meaning only when predicted probabilities are well-calibrated
📐 Mathematical Derivations
Decision Theoretic Formulation: Let be the cost of a false alarm and be the cost of a missed detection (assuming zero cost for correct predictions). The expected cost of predicting positive given calibrated probability is: . The expected cost of predicting negative is: .
Predict if , yielding the optimal threshold: . When missed detection is 9 times costlier than a false alarm (), .
Predict if , yielding the optimal threshold: . When missed detection is 9 times costlier than a false alarm (), .
🏭 Production Trade-offs
Business application archetypes: ① Recall-Dominant (Low $\tau$): Fraud, safety alarms, cancer screening—cost of FN is catastrophic; human reviewers handle FPs. ② Precision-Dominant (High $\tau$): Push notifications, automated loan grants, autonomous actions—interrupting users or erroneous automated payouts destroys business value. ③ Quantile/Capacity-Constrained: Select the top items daily based on human operational capacity.
⚠️ Common Interview Pitfalls
- ✕Tuning the decision threshold on the test set, creating optimistic selection leakage
- ✕Applying theoretical cost thresholds to raw uncalibrated probabilities from deep neural networks or tree ensembles
🎯 Interviewer Follow-up Inquiries
- ?How does probability calibration impact the validity of the Bayes optimal threshold formula ?
- ?How do you handle threshold drift when the online positive class prevalence shifts dynamically?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚