M2-081M2: Classical Machine LearningModel CalibrationEasy
⚡Flashcard Drill➔Mastery:

Defining Model Calibration: Why High Accuracy Does Not Imply Good Calibration

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-082 →
📐 Mathematical Definition
E[Y∣p^=p]=p\mathbb E[Y\mid \hat p=p]=p
⚡ One-Sentence Core Answer
A model is calibrated when its predicted probability matches empirical ground truth frequency; a highly accurate model can still be severely overconfident.

📌 Key Takeaways

  • •
    Definition: P(Y=1∣P^=p)=pP(Y = 1 \mid \hat{P} = p) = p for all confidence levels p∈[0,1]p \in [0, 1]
  • •
    Accuracy depends only on ranking/argmax; calibration evaluates exact posterior probability correctness
  • •
    Modern deep networks achieve high accuracy but suffer from overconfidence due to cross-entropy over-optimization

📐 Mathematical Derivations

Formal Definition: A probabilistic classifier P^(Y=1∣X)\hat{P}(Y=1|X) is perfectly calibrated if: P(Y=1∣P^(Y=1∣X)=p)=p,∀p∈[0,1]P(Y = 1 \mid \hat{P}(Y=1|X) = p) = p, \quad \forall p \in [0, 1].
Accuracy vs Calibration Separation: Accuracy depends solely on whether P^(Y=1∣X)>0.5\hat{P}(Y=1|X) > 0.5. Consider two models on 100 samples:
- Model A: Predicts p=0.99p=0.99 for all 80 true positives and p=0.99p=0.99 for all 20 false positives. Accuracy = 80%80\%. However, among samples predicted with 99%99\% confidence, actual accuracy is only 80%80\%. It is dangerously overconfident.
- Model B: Predicts p=0.80p=0.80 for all 100 samples. Accuracy = 80%80\%. Here P(Y=1∣P^=0.80)=0.80P(Y=1|\hat{P}=0.80) = 0.80, making it perfectly calibrated.
Ranking metrics (ROC-AUC) are invariant to monotonic transformations, whereas calibration changes completely under monotonic scaling.

🏭 Production Trade-offs

When calibration is essential: ① Risk-Sensitive Decisions: Medical diagnoses, autonomous driving emergency braking, and insurance pricing where expected costs depend on exact probabilities. ② Downstream Multi-stage Ensembles: Blending probabilities from diverse models requires unified calibration. ③ Bandits & RL: Thompson Sampling depends on posterior probability fidelity.
⚠️ Common Interview Pitfalls
  • ✕
    Using uncalibrated soft probabilities from deep neural networks as confidence estimates in safety-critical pipelines
  • ✕
    Assuming that achieving an ROC-AUC of 0.95 means model probabilities reflect real-world event frequencies
🎯 Interviewer Follow-up Inquiries
  • ?
    Why do modern deep neural networks with batch normalization and weight decay tend to be more overconfident than older networks?
  • ?
    How does Brier score evaluate calibration and refinement simultaneously?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-080: Why Global Normalization Before Splitting Causes Data Leakage and How to Fix It📋 Back to BankNext →M2-082: Comparing Platt Scaling and Isotonic Regression for Probability Calibration