O6EasyPart O · BA-to-Python Data Science & Stats Kernels

Robust Data Contract Gate (NaN Rate + Median-MAD Z-Score)

⚡Flashcard Drill➔

Build an outlier-resistant ingestion quality gate combining null-rate checks with a 50%-breakdown Median-MAD robust Z-score.

⏱️ Time: O(N)
💾 Space: O(N)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def validate_feature_batch(values: np.ndarray, max_null_rate: float = 0.05, z_thresh: float = 6.0) -> dict:
n = len(values)
if n == 0:
return {"passed": False, "null_rate": 1.0, "outlier_count": 0, "reason": "empty_batch"}
nan_mask =
null_rate = float(np.mean(nan_mask))
valid = values[~nan_mask]
if null_rate > max_null_rate or len(valid) == 0:
return {"passed": False, "null_rate": null_rate, "outlier_count": 0, "reason": "null_rate_exceeded"}
med = float(np.median(valid))
mad = float(np.median(np.abs(valid - med)))
if mad < 1e-12:
outliers = int(np.sum(np.abs(valid - med) > 1e-6))
else:
robust_z = 0.6745 * np.abs(valid - med) / mad
outliers =
passed = outliers <= max(1, int(0.01 * len(valid)))
return {"passed": passed, "null_rate": null_rate, "outlier_count": outliers, "reason": "ok" if passed else "outlier_spike"}
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why does standard 3-sigma fail on production telemetry spikes?
Q2: What happens in CI/CD when a data contract gate trips?
Next: Training-Serving Skew Audit via Two-Sample KS Test →
💡

Core Mental Anchor / Mnemonic

Filter NaNs first, use Median-MAD for 50% breakdown resistance

📐 Mathematical Derivation & Core Formula

$$\text{MAD} = \text{median}(|x - \tilde{x}|),\quad z_{\text{robust}} = \frac{0.6745(x - \tilde{x})}{\text{MAD}}$$
Classical mean/std has a 0% breakdown point: a single corrupted 1e6 value masks outliers by inflating std. Scaling absolute median deviation by 0.6745 gives a consistent Gaussian-equivalent robust Z-score immune to heavy batch corruption.

🔄 Tensor Dimensions & Shape Flow

values: (N,) -> valid non-NaN -> median/MAD -> boolean contract gate

🛡️ Numerical Stability & Pitfalls Checklist

  • Compute null rates and slice ~np.isnan before calling np.median so NaNs do not poison aggregates
  • Handle zero-MAD constant distributions explicitly to avoid division by zero
  • Hold previous healthy snapshots and quarantine failing batches rather than overwriting serving tables

💻 Reference Code Implementation

import numpy as np

def validate_feature_batch(values: np.ndarray, max_null_rate: float = 0.05, z_thresh: float = 6.0) -> dict:
    n = len(values)
    if n == 0:
        return {"passed": False, "null_rate": 1.0, "outlier_count": 0, "reason": "empty_batch"}
    nan_mask = np.isnan(values)
    null_rate = float(np.mean(nan_mask))
    valid = values[~nan_mask]
    if null_rate > max_null_rate or len(valid) == 0:
        return {"passed": False, "null_rate": null_rate, "outlier_count": 0, "reason": "null_rate_exceeded"}
    med = float(np.median(valid))
    mad = float(np.median(np.abs(valid - med)))
    if mad < 1e-12:
        outliers = int(np.sum(np.abs(valid - med) > 1e-6))
    else:
        robust_z = 0.6745 * np.abs(valid - med) / mad
        outliers = int(np.sum(robust_z > z_thresh))
    passed = outliers <= max(1, int(0.01 * len(valid)))
    return {"passed": passed, "null_rate": null_rate, "outlier_count": outliers, "reason": "ok" if passed else "outlier_spike"}

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

clean = np.concatenate([np.ones(98), [np.nan, 100.0]])
r1 = validate_feature_batch(clean)
assert r1['passed'] is True
bad = np.concatenate([np.zeros(90), np.full(10, 50.0)])
r2 = validate_feature_batch(bad)
assert r2['passed'] is False and r2['reason'] == 'outlier_spike'
print('O6 PASS')

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why does standard 3-sigma fail on production telemetry spikes?
Even a small cluster of 1000x unit-mismatch rows inflates sample variance enough to keep their own standard Z-scores deceptively small.
Q2:What happens in CI/CD when a data contract gate trips?
Halt downstream feature materialization or retraining, keep yesterday's validated snapshot live, and route failing partitions to quarantine.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: K-Means++ D-Squared Initialization for RFM & CodebooksBack to Part O & All 105Next: Training-Serving Skew Audit via Two-Sample KS Test →