Probability Calibration

Probability calibration turns model scores into true interpretable probabilities. Platt scaling fits a logistic regression on the raw score: $P(y=1|x) = \frac{1}{1+\exp(Af(x)+B)}$, with parameters $A$ and $B$ estimated by maximum likelihood on a separate validation set — suitable for models whose scores are monotonically but sigmoidally distorted (SVM, GBDT). Isotonic regression fits a non-parametric monotone mapping on the validation set; it is more flexible but needs more data and overfits more easily. Evaluation tools: the Brier score $BS = \frac{1}{N}\sum_{i=1}^{N}(p_i - y_i)^2$ (lower is better) and reliability diagrams (bucket predictions by probability and compare average prediction with actual frequency).

decisions that depend on the probability value itself (credit thresholds, risk scorecards, medical diagnosis), merging probabilities from multiple models, cost-sensitive decisions; interviews ask about “calibration vs AUC”, “Platt vs Isotonic”, and “why logistic regression is naturally calibrated”.

AUC only measures ranking ability, not the numerical correctness of probabilities — a model can say 0.9 while only 60% of such cases are positive; Platt/Isotonic map scores to decision-ready probabilities. Key property: calibration is a monotonic (rank-preserving) transform, so it does not change AUC — good calibration and good discrimination are orthogonal and must be evaluated separately.

Test Your Knowledge: Practice Questions for "Probability Calibration"

Single choice pitfall questions with instant feedback and mistake tracking.