M2-118M2: Classical Machine LearningEvaluation Metrics & Hyperparameter TuningMedium
⚡Flashcard Drill➔Mastery:

Multi-Metric Trade-offs and Designing an Overall Evaluation Criterion (OEC)

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-119 →
📐 Mathematical Definition
OEC=∑kwk⋅normalized(metrick)\text{OEC}=\sum_k w_k\cdot\text{normalized}(\text{metric}_k)
⚡ One-Sentence Core Answer
An OEC synthesizes competing metrics into a unified objective to streamline decision-making, bounded by non-negotiable guardrail metrics.

📌 Key Takeaways

  • •
    Single decision target: OEC=∑wk⋅Normalized(Metrick)\text{OEC} = \sum w_k \cdot \text{Normalized}(\text{Metric}_k) prevents p-hacking across multiple readouts
  • •
    Guardrail metrics: hard constraints (e.g., p99 latency ≤200ms\le 200\text{ms}, return rate ≤3%\le 3\%) that invalidate launches if breached
  • •
    Pre-commitment: weights and metric normalizations must be mathematically locked prior to running experiments

📐 Mathematical Derivations

Mathematical Formulation: In online experiments, interventions affect multiple metrics simultaneously (e.g., CTR increases by +2%+2\%, but app uninstalls increase by +1%+1\%).
Construct the Overall Evaluation Criterion (OEC) as: OEC=∑k=1Kwk(Mktreatment−MkcontrolMkcontrol)×1σk\text{OEC} = \sum_{k=1}^K w_k \left( \frac{M_k^{\text{treatment}} - M_k^{\text{control}}}{M_k^{\text{control}}} \right) \times \frac{1}{\sigma_k}, or translate all metrics into equivalent monetary value: OEC=ΔRevenue−λ⋅ΔSupport Costs\text{OEC} = \Delta \text{Revenue} - \lambda \cdot \Delta \text{Support Costs}.
Guardrail Constraints: Formulate decision as a constrained optimization problem:
max⁡OECs.t.Gj≤Cj,  ∀j∈{1,…,J}\max \text{OEC} \quad \text{s.t.} \quad G_j \le C_j, \; \forall j \in \{1, \dots, J\}. Even if OEC rises significantly, violating a guardrail (such as crash rate or fairness constraint) automatically halts rollout.

🏭 Production Trade-offs

Organizational governance: Pre-defining OEC weights eliminates internal debates and post-hoc rationalization. However, an OEC must exclude hard safety and technical reliability metrics, which should always be treated as independent pass/fail guardrails.
⚠️ Common Interview Pitfalls
  • ✕
    Adjusting OEC weights after inspecting A/B test results (a form of p-hacking and confirmation bias)
  • ✕
    Incorporating guardrail metrics directly into the OEC, allowing severe latency or crash degradation to be masked by click gains
🎯 Interviewer Follow-up Inquiries
  • ?
    How do you establish conversion exchange rates between disparate user metrics (e.g., clicks vs uninstalls) when designing an OEC?
  • ?
    Why does monitoring 20 independent metrics without multiplicity correction inflate family-wise error rate to over 64%?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-117: Preventing Data Leakage in Time Series and Panel Data: Critical Safeguards📋 Back to BankNext →M2-119: Automated Hyperparameter Optimization Tools and Industrial Best-Practice Workflows