P6MediumPart P · MLE Production Engineering & Calibration

Platt Scaling & Expected Calibration Error (ECE) for Ads pCTR

⚡Flashcard Drill➔

Implement Expected Calibration Error (ECE) and NLL-fitted Platt Scaling to calibrate uncalibrated logits back into true bid-safe probabilities.

⏱️ Time: O(S * N)
💾 Space: O(N)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def expected_calibration_error(probs: np.ndarray, labels: np.ndarray, n_bins: int = 10) -> float:
bins =
ece = 0.0
n = len(probs)
for i in range(n_bins):
if i == n_bins - 1:
mask = (probs >= bins[i]) & (probs <= bins[i + 1])
else:
mask = (probs >= bins[i]) & (probs < bins[i + 1])
cnt = int(np.sum(mask))
if cnt > 0:
acc = float(np.mean(labels[mask]))
conf = float(np.mean(probs[mask]))
ece += (cnt / n) * abs(acc - conf)
return float(ece)
def fit_platt_scaling(logits: np.ndarray, labels: np.ndarray, lr: float = 0.1, steps: int = 300) -> tuple[float, float]:
a, b = 1.0, 0.0
for _ in range(steps):
z = np.clip(a * logits + b, -30.0, 30.0)
p = 1.0 / (1.0 + np.exp(-z))
err =
a -= lr * float(np.mean(err * logits))
b -= lr * float(np.mean(err))
return a, b
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How do you undo 1:10 negative downsampling when serving pCTR?
Q2: Compare Platt Scaling vs Isotonic Regression for calibration.
Next: GPU Roofline Arithmetic Intensity & Ridge-Point Analyzer →
💡

Core Mental Anchor / Mnemonic

AUC measures rank, ECE validates probability magnitude for eCPM bidding

📐 Mathematical Derivation & Core Formula

$$\hat{p} = \sigma(a \cdot z + b),\quad \text{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n} |\text{acc}(B_m) - \text{conf}(B_m)|$$
Because ad auctions compute eCPM=pCTR×Bid\text{eCPM} = \text{pCTR} \times \text{Bid}, monotonic AUC alone is insufficient. Fitting p^=σ(az+b)\hat{p} = \sigma(az + b) on a held-out calibration split corrects temperature overconfidence and bias, verifiable via bucketed ∣acc−conf∣|\text{acc} - \text{conf}| ECE.

🔄 Tensor Dimensions & Shape Flow

logits: (N,), labels: (N,) -> ECE scalar & (a, b) calibration parameters

🛡️ Numerical Stability & Pitfalls Checklist

  • Fit calibration parameters strictly on a held-out calibration split rather than training rows
  • Include exact 1.0 in the rightmost ECE bin
  • Clip pre-sigmoid logits within [-30, 30] to prevent exp() overflow

💻 Reference Code Implementation

import numpy as np

def expected_calibration_error(probs: np.ndarray, labels: np.ndarray, n_bins: int = 10) -> float:
    bins = np.linspace(0.0, 1.0, n_bins + 1)
    ece = 0.0
    n = len(probs)
    for i in range(n_bins):
        if i == n_bins - 1:
            mask = (probs >= bins[i]) & (probs <= bins[i + 1])
        else:
            mask = (probs >= bins[i]) & (probs < bins[i + 1])
        cnt = int(np.sum(mask))
        if cnt > 0:
            acc = float(np.mean(labels[mask]))
            conf = float(np.mean(probs[mask]))
            ece += (cnt / n) * abs(acc - conf)
    return float(ece)

def fit_platt_scaling(logits: np.ndarray, labels: np.ndarray, lr: float = 0.1, steps: int = 300) -> tuple[float, float]:
    a, b = 1.0, 0.0
    for _ in range(steps):
        z = np.clip(a * logits + b, -30.0, 30.0)
        p = 1.0 / (1.0 + np.exp(-z))
        err = p - labels
        a -= lr * float(np.mean(err * logits))
        b -= lr * float(np.mean(err))
    return a, b

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

rng = np.random.default_rng(10)
logits = rng.normal(0, 2.0, 2000)
true_p = 1.0 / (1.0 + np.exp(-(0.5 * logits - 0.8)))
y = (rng.random(2000) < true_p).astype(np.float64)
uncal_p = 1.0 / (1.0 + np.exp(-logits))
ece_before = expected_calibration_error(uncal_p, y)
a, b = fit_platt_scaling(logits, y, lr=0.2, steps=400)
cal_p = 1.0 / (1.0 + np.exp(-(a * logits + b)))
ece_after = expected_calibration_error(cal_p, y)
assert ece_after < 0.03 and ece_after < ece_before * 0.25
print('P6 PASS')

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How do you undo 1:10 negative downsampling when serving pCTR?
Shift the predicted log-odds by adding log(negative_sample_rate): p_real = q / (q + (1 - q) / w).
Q2:Compare Platt Scaling vs Isotonic Regression for calibration.
Platt only fits 2 parameters (a, b), making it low-variance for small splits; Isotonic fits a non-parametric monotonic step curve that handles non-sigmoidal distortions when ample validation data exists.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Beta-Bernoulli Thompson Sampling for Cold-Start & RoutingBack to Part P & All 105Next: GPU Roofline Arithmetic Intensity & Ridge-Point Analyzer →