M2-086M2: Classical Machine LearningEvaluation Metrics & Hyperparameter TuningEasy
⚡Flashcard Drill➔Mastery:

Common Evaluation Metrics for Classification, Regression, and Ranking

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-087 →
📐 Mathematical Definition
F1=2PRP+R,RMSE=1n∑(y−y^)2F_1=\frac{2PR}{P+R},\qquad \mathrm{RMSE}=\sqrt{\frac1n\sum(y-\hat y)^2}
⚡ One-Sentence Core Answer
Classification uses Accuracy, F1, PR-AUC, and ROC-AUC; regression uses RMSE, MAE, and MAPE; ranking uses NDCG, MRR, and MAP.

📌 Key Takeaways

  • •
    Classification: Accuracy, Precision, Recall, F1, PR-AUC, ROC-AUC, Brier score
  • •
    Regression: RMSE (penalizes large errors), MAE (robust to outliers), MAPE (scale-invariant relative error)
  • •
    Ranking: NDCG (graded relevance with logarithmic discount), MRR (position of first hit), MAP (mean precision at relevant ranks)

📐 Mathematical Derivations

Mathematical Formulations:
① Ranking — NDCG@K: DCG@K=∑i=1K2ri−1log⁡2(i+1)\text{DCG}@K = \sum_{i=1}^K \frac{2^{r_i} - 1}{\log_2(i + 1)}, NDCG@K=DCG@KIDCG@K\text{NDCG}@K = \frac{\text{DCG}@K}{\text{IDCG}@K}, where IDCG@K\text{IDCG}@K is the ideal DCG obtained by sorting items by ground truth relevance rir_i.
② Ranking — MRR (Mean Reciprocal Rank): MRR=1∣Q∣∑q=1∣Q∣1rankq\text{MRR} = \frac{1}{|Q|} \sum_{q=1}^{|Q|} \frac{1}{\text{rank}_q}, measuring the reciprocal rank of the first relevant item.
③ Regression — Metrics: RMSE=1N∑(yi−y^i)2\text{RMSE} = \sqrt{\frac{1}{N}\sum (y_i - \hat{y}_i)^2} (optimizes conditional mean); MAE=1N∑∣yi−y^i∣\text{MAE} = \frac{1}{N}\sum |y_i - \hat{y}_i| (optimizes conditional median); Huber loss smoothly transitions between quadratic and linear regimes.
④ Classification — Matthews Correlation Coefficient (MCC): MCC=TP⋅TN−FP⋅FN(TP+FP)(TP+FN)(TN+FP)(TN+FN)\text{MCC} = \frac{\text{TP} \cdot \text{TN} - \text{FP} \cdot \text{FN}}{\sqrt{(\text{TP}+\text{FP})(\text{TP}+\text{FN})(\text{TN}+\text{FP})(\text{TN}+\text{FN})}}, symmetric and robust to extreme imbalance.

🏭 Production Trade-offs

Metric alignment with business goals: In e-commerce search, NDCG@10 reflects top-of-page user satisfaction. In supply chain inventory forecasting, asymmetric pinball loss (quantile loss) is preferred over RMSE because stockouts are costlier than excess holding costs.
⚠️ Common Interview Pitfalls
  • ✕
    Using MAPE when actual values can be zero or near-zero, producing division-by-zero explosions
  • ✕
    Evaluating search ranking using binary accuracy or unweighted recall instead of position-discounted NDCG
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does minimizing MSE regression loss predict the conditional mean E[Y∣X]\mathbb{E}[Y|X] while MAE predicts the conditional median?
  • ?
    What are the structural differences between MAP (Mean Average Precision) and NDCG@K in recommendation systems?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-085: Scenarios Where Calibration is Critical: Core Use Cases and Concrete Examples📋 Back to BankNext →M2-087: Comparing Grid Search, Random Search, and Bayesian Optimization