M2-086M2: Classical Machine LearningEvaluation Metrics & Hyperparameter TuningEasy
⚡Flashcard Drill➔Mastery:
Common Evaluation Metrics for Classification, Regression, and Ranking
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Classification uses Accuracy, F1, PR-AUC, and ROC-AUC; regression uses RMSE, MAE, and MAPE; ranking uses NDCG, MRR, and MAP.
📌 Key Takeaways
- •Classification: Accuracy, Precision, Recall, F1, PR-AUC, ROC-AUC, Brier score
- •Regression: RMSE (penalizes large errors), MAE (robust to outliers), MAPE (scale-invariant relative error)
- •Ranking: NDCG (graded relevance with logarithmic discount), MRR (position of first hit), MAP (mean precision at relevant ranks)
📐 Mathematical Derivations
Mathematical Formulations:
① Ranking — NDCG@K: , , where is the ideal DCG obtained by sorting items by ground truth relevance .
② Ranking — MRR (Mean Reciprocal Rank): , measuring the reciprocal rank of the first relevant item.
③ Regression — Metrics: (optimizes conditional mean); (optimizes conditional median); Huber loss smoothly transitions between quadratic and linear regimes.
④ Classification — Matthews Correlation Coefficient (MCC): , symmetric and robust to extreme imbalance.
① Ranking — NDCG@K: , , where is the ideal DCG obtained by sorting items by ground truth relevance .
② Ranking — MRR (Mean Reciprocal Rank): , measuring the reciprocal rank of the first relevant item.
③ Regression — Metrics: (optimizes conditional mean); (optimizes conditional median); Huber loss smoothly transitions between quadratic and linear regimes.
④ Classification — Matthews Correlation Coefficient (MCC): , symmetric and robust to extreme imbalance.
🏭 Production Trade-offs
Metric alignment with business goals: In e-commerce search, NDCG@10 reflects top-of-page user satisfaction. In supply chain inventory forecasting, asymmetric pinball loss (quantile loss) is preferred over RMSE because stockouts are costlier than excess holding costs.
⚠️ Common Interview Pitfalls
- ✕Using MAPE when actual values can be zero or near-zero, producing division-by-zero explosions
- ✕Evaluating search ranking using binary accuracy or unweighted recall instead of position-discounted NDCG
🎯 Interviewer Follow-up Inquiries
- ?Why does minimizing MSE regression loss predict the conditional mean while MAE predicts the conditional median?
- ?What are the structural differences between MAP (Mean Average Precision) and NDCG@K in recommendation systems?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚