PR-AUC vs ROC & Threshold Tuning

PR-AUC Superiority in Imbalance & Optimal Threshold Calibration formalizes classification evaluation and operating point selection under severe positive rarity (e.g. positive prevalence $<0.1\%$); while ROC Curves plot TPR vs FPR (where the massive true-negative denominator $N$ artificially suppresses FPR, inflating ROC-AUC to deceptively high $>0.95$), Precision-Recall (PR) Curves plot Precision vs Recall, completely excluding the uninformative true-negative denominator to expose precision collapse; furthermore, optimal decision threshold calibration dynamically sweeps cutoffs $T^* \in (0, 1)$ to maximize F1-score, Youden's Index ($TPR - FPR$), or minimize expected monetary loss using business cost matrices $C_{\text{FP}}$ and $C_{\text{FN}}$.

Payment fraud detection, rare pathology diagnosis, ad conversion prediction, and cyber-intrusion alerts.

Default 0.5 probability thresholds predict zero fraud, while ROC-AUC masks catastrophic false-alarm rates; PR-AUC and cost-calibrated thresholding deliver mathematically grounded production decision cutoffs.

Mathematically prove why ROC curves are invariant to class prevalence shifts while PR curves reflect true positive ratio changes?

Contrast random baseline performance: ROC-AUC is always 0.5, whereas PR-AUC baseline equals the true positive class prevalence $P/(P+N)$?

Compare threshold selection criteria: Max F1 vs Youden's Index vs Fixed-Specificity constraint (e.g. FPR $\le 0.001$)?

How to use Isotonic Regression to calibrate raw model scores into true empirical probabilities for downstream dollar-cost decision making?

Why does linear trapezoidal integration overestimate PR-AUC, mandating interpolated Average Precision (AP) sums?