Probability Calibration

Probability calibration turns model scores into true interpretable probabilities. Platt scaling fits a logistic regression on the raw score: $P(y=1|x) = \frac{1}{1+\exp(Af(x)+B)}$, with parameters $A$ and $B$ estimated by maximum likelihood on a separate validation set — suitable for models whose scores are monotonically but sigmoidally distorted (SVM, GBDT). Isotonic regression fits a non-parametric monotone mapping on the validation set; it is more flexible but needs more data and overfits more easily. Evaluation tools: the Brier score $BS = \frac{1}{N}\sum_{i=1}^{N}(p_i - y_i)^2$ (lower is better) and reliability diagrams (bucket predictions by probability and compare average prediction with actual frequency).

decisions that depend on the probability value itself (credit thresholds, risk scorecards, medical diagnosis), merging probabilities from multiple models, cost-sensitive decisions; interviews ask about “calibration vs AUC”, “Platt vs Isotonic”, and “why logistic regression is naturally calibrated”.

AUC only measures ranking ability, not the numerical correctness of probabilities — a model can say 0.9 while only 60% of such cases are positive; Platt/Isotonic map scores to decision-ready probabilities. Key property: calibration is a monotonic (rank-preserving) transform, so it does not change AUC — good calibration and good discrimination are orthogonal and must be evaluated separately.

Write Platt's formula $P(y=1|x) = \frac{1}{1+\exp(Af(x)+B)}$ and explain how A and B are fit. Is it the same thing as logistic regression?

Compare Platt vs Isotonic: assumptions, data requirements, overfitting risk, and which models each suits?

Write the Brier score $BS = \frac{1}{N}\sum (p_i - y_i)^2$; give examples of “good discrimination, bad calibration” and the reverse?

Why does calibration leave AUC unchanged? How does a rank-preserving monotone transform preserve AUC?

How to read a reliability diagram? Why is logistic regression naturally calibrated while SVM/GBDT need calibration?