M2-074M2: Classical Machine LearningClass Imbalance LearningMedium
⚡Flashcard Drill➔Mastery:

Classification Threshold Selection Strategies Across Diverse Business Objectives

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-075 →
📐 Mathematical Definition
threshold∗=arg⁡min⁡τ [cFPFP(τ)+cFNFN(τ)]\text{threshold}^*=\arg\min_\tau\ \big[c_{FP}\mathrm{FP}(\tau)+c_{FN}\mathrm{FN}(\tau)\big]
⚡ One-Sentence Core Answer
Select thresholds by minimizing expected cost via cost matrices or meeting business constraints (e.g., recall floor) using a calibrated validation set.

📌 Key Takeaways

  • •
    Cost minimization: select τ∗=arg⁡min⁡τ[cFP⋅FP(τ)+cFN⋅FN(τ)]\tau^* = \arg\min_\tau [c_{\text{FP}} \cdot \text{FP}(\tau) + c_{\text{FN}} \cdot \text{FN}(\tau)]
  • •
    Constraint optimization: maximize Precision subject to Recall≥90%\text{Recall} \ge 90\% (or vice-versa)
  • •
    Probability calibration: thresholds have stable business meaning only when predicted probabilities are well-calibrated

📐 Mathematical Derivations

Decision Theoretic Formulation: Let cFPc_{\text{FP}} be the cost of a false alarm and cFNc_{\text{FN}} be the cost of a missed detection (assuming zero cost for correct predictions). The expected cost of predicting positive given calibrated probability p=P(y=1∣x)p = P(y=1|x) is: E[Cost∣y^=1]=(1−p)cFP\mathbb{E}[\text{Cost}|\hat{y}=1] = (1 - p) c_{\text{FP}}. The expected cost of predicting negative is: E[Cost∣y^=0]=p cFN\mathbb{E}[\text{Cost}|\hat{y}=0] = p \, c_{\text{FN}}.
Predict y^=1\hat{y}=1 if E[Cost∣y^=1]<E[Cost∣y^=0]\mathbb{E}[\text{Cost}|\hat{y}=1] < \mathbb{E}[\text{Cost}|\hat{y}=0], yielding the optimal threshold: τ∗=cFPcFP+cFN\tau^* = \frac{c_{\text{FP}}}{c_{\text{FP}} + c_{\text{FN}}}. When missed detection is 9 times costlier than a false alarm (cFN=9cFPc_{\text{FN}} = 9 c_{\text{FP}}), τ∗=0.10\tau^* = 0.10.

🏭 Production Trade-offs

Business application archetypes: ① Recall-Dominant (Low $\tau$): Fraud, safety alarms, cancer screening—cost of FN is catastrophic; human reviewers handle FPs. ② Precision-Dominant (High $\tau$): Push notifications, automated loan grants, autonomous actions—interrupting users or erroneous automated payouts destroys business value. ③ Quantile/Capacity-Constrained: Select the top KK items daily based on human operational capacity.
⚠️ Common Interview Pitfalls
  • ✕
    Tuning the decision threshold on the test set, creating optimistic selection leakage
  • ✕
    Applying theoretical cost thresholds to raw uncalibrated probabilities from deep neural networks or tree ensembles
🎯 Interviewer Follow-up Inquiries
  • ?
    How does probability calibration impact the validity of the Bayes optimal threshold formula τ∗=cFP/(cFP+cFN)\tau^* = c_{\text{FP}} / (c_{\text{FP}} + c_{\text{FN}})?
  • ?
    How do you handle threshold drift when the online positive class prevalence shifts dynamically?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-073: Why Prioritize PR-AUC Over ROC-AUC in Highly Imbalanced Regimes?📋 Back to BankNext →M2-075: Cost-Sensitive Learning: Implementation and Relationship with Re-sampling