P6MediumPart P · MLE Production Engineering & Calibration
Platt Scaling & Expected Calibration Error (ECE) for Ads pCTR
Implement Expected Calibration Error (ECE) and NLL-fitted Platt Scaling to calibrate uncalibrated logits back into true bid-safe probabilities.
⏱️ Time:
O(S * N)💾 Space:
O(N)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def expected_calibration_error(probs: np.ndarray, labels: np.ndarray, n_bins: int = 10) -> float:
bins =
ece = 0.0
n = len(probs)
for i in range(n_bins):
if i == n_bins - 1:
mask = (probs >= bins[i]) & (probs <= bins[i + 1])
else:
mask = (probs >= bins[i]) & (probs < bins[i + 1])
cnt = int(np.sum(mask))
if cnt > 0:
acc = float(np.mean(labels[mask]))
conf = float(np.mean(probs[mask]))
ece += (cnt / n) * abs(acc - conf)
return float(ece)
def fit_platt_scaling(logits: np.ndarray, labels: np.ndarray, lr: float = 0.1, steps: int = 300) -> tuple[float, float]:
a, b = 1.0, 0.0
for _ in range(steps):
z = np.clip(a * logits + b, -30.0, 30.0)
p = 1.0 / (1.0 + np.exp(-z))
err =
a -= lr * float(np.mean(err * logits))
b -= lr * float(np.mean(err))
return a, b
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How do you undo 1:10 negative downsampling when serving pCTR?
Q2: Compare Platt Scaling vs Isotonic Regression for calibration.
💡
Core Mental Anchor / Mnemonic
AUC measures rank, ECE validates probability magnitude for eCPM bidding
📐 Mathematical Derivation & Core Formula
$$\hat{p} = \sigma(a \cdot z + b),\quad \text{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n} |\text{acc}(B_m) - \text{conf}(B_m)|$$
Because ad auctions compute , monotonic AUC alone is insufficient. Fitting on a held-out calibration split corrects temperature overconfidence and bias, verifiable via bucketed ECE.
🔄 Tensor Dimensions & Shape Flow
logits: (N,), labels: (N,) -> ECE scalar & (a, b) calibration parameters
🛡️ Numerical Stability & Pitfalls Checklist
- Fit calibration parameters strictly on a held-out calibration split rather than training rows
- Include exact 1.0 in the rightmost ECE bin
- Clip pre-sigmoid logits within [-30, 30] to prevent exp() overflow
💻 Reference Code Implementation
import numpy as np
def expected_calibration_error(probs: np.ndarray, labels: np.ndarray, n_bins: int = 10) -> float:
bins = np.linspace(0.0, 1.0, n_bins + 1)
ece = 0.0
n = len(probs)
for i in range(n_bins):
if i == n_bins - 1:
mask = (probs >= bins[i]) & (probs <= bins[i + 1])
else:
mask = (probs >= bins[i]) & (probs < bins[i + 1])
cnt = int(np.sum(mask))
if cnt > 0:
acc = float(np.mean(labels[mask]))
conf = float(np.mean(probs[mask]))
ece += (cnt / n) * abs(acc - conf)
return float(ece)
def fit_platt_scaling(logits: np.ndarray, labels: np.ndarray, lr: float = 0.1, steps: int = 300) -> tuple[float, float]:
a, b = 1.0, 0.0
for _ in range(steps):
z = np.clip(a * logits + b, -30.0, 30.0)
p = 1.0 / (1.0 + np.exp(-z))
err = p - labels
a -= lr * float(np.mean(err * logits))
b -= lr * float(np.mean(err))
return a, b🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
rng = np.random.default_rng(10)
logits = rng.normal(0, 2.0, 2000)
true_p = 1.0 / (1.0 + np.exp(-(0.5 * logits - 0.8)))
y = (rng.random(2000) < true_p).astype(np.float64)
uncal_p = 1.0 / (1.0 + np.exp(-logits))
ece_before = expected_calibration_error(uncal_p, y)
a, b = fit_platt_scaling(logits, y, lr=0.2, steps=400)
cal_p = 1.0 / (1.0 + np.exp(-(a * logits + b)))
ece_after = expected_calibration_error(cal_p, y)
assert ece_after < 0.03 and ece_after < ece_before * 0.25
print('P6 PASS')🎯 Core Follow-up Inquiries & Reference Answers
Q1:How do you undo 1:10 negative downsampling when serving pCTR?
Shift the predicted log-odds by adding log(negative_sample_rate): p_real = q / (q + (1 - q) / w).
Q2:Compare Platt Scaling vs Isotonic Regression for calibration.
Platt only fits 2 parameters (a, b), making it low-variance for small splits; Isotonic fits a non-parametric monotonic step curve that handles non-sigmoidal distortions when ample validation data exists.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.