M2-087M2: Classical Machine LearningEvaluation Metrics & Hyperparameter TuningMedium
⚡Flashcard Drill➔Mastery:

Comparing Grid Search, Random Search, and Bayesian Optimization

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-088 →
📐 Mathematical Definition
BO: max⁡ EI(x)=E[max⁡(0,f(x)−f∗)]\text{BO}:\ \max\ \mathrm{EI}(x)=\mathbb E[\max(0,f(x)-f^*)]
⚡ One-Sentence Core Answer
Grid search suffers from exponential dimensional explosion; Random search explores continuous spaces efficiently; Bayesian optimization constructs probabilistic surrogates to balance exploration and exploitation.

📌 Key Takeaways

  • •
    Grid search: exhaustive evaluation, catastrophic O(kd)O(k^d) exponential scaling with dimension dd
  • •
    Random search: statistically superior under low effective dimensionality, samples distinct values on every trial
  • •
    Bayesian optimization (GP/TPE): models P(y∣x)P(y|x) with Gaussian Processes or Parzen Estimators, guided by Acquisition Functions (EI, UCB)

📐 Mathematical Derivations

Theoretical comparisons: ① Why Random Search beats Grid Search: In typical hyperparameter tuning, only a small subset of hyperparameters have high effective importance (low intrinsic dimensionality). In a 3×33 \times 3 grid with 9 evaluations, only 3 distinct values are tested for each parameter. Random search tests 9 distinct values for every parameter across the same 9 trials, achieving triple the coverage density on the critical dimensions. ② Bayesian Optimization Framework: Given prior observations D1:t−1={(xi,yi)}\mathcal{D}_{1:t-1} = \{(x_i, y_i)\}, fit surrogate model f(x)∼GP(μ(x),k(x,x′))f(x) \sim \mathcal{GP}(\mu(x), k(x, x')). Compute the next point using an Acquisition Function such as Expected Improvement (EI): EI(x)=E[max⁡(0,f(x)−f(x+))]=(μ(x)−f(x+))Φ(Z)+σ(x)ϕ(Z)\text{EI}(x) = \mathbb{E}[\max(0, f(x) - f(x^+))] = (\mu(x) - f(x^+)) \Phi(Z) + \sigma(x) \phi(Z), where Z=μ(x)−f(x+)σ(x)Z = \frac{\mu(x) - f(x^+)}{\sigma(x)}. μ(x)\mu(x) drives exploitation, while uncertainty σ(x)\sigma(x) drives exploration.

🏭 Production Trade-offs

Practical tooling choice: ① Few iterations / cheap models: Random search is trivially parallelizable across clusters without synchronization overhead. ② Expensive models (Deep Learning / large tabular): Bayesian Optimization (Optuna TPE / Ray Tune) converges in significantly fewer trials. ③ Resource allocation: Combine Bayesian optimization with early-stopping pruning algorithms like Hyperband / ASHA.
⚠️ Common Interview Pitfalls
  • ✕
    Running high-dimensional grid search across 6+ hyperparameters, wasting compute on unpromising combinations
  • ✕
    Using Gaussian Process Bayesian Optimization in high dimensions (d>30d > 30) or with thousands of trials, where GP cubic complexity O(n3)O(n^3) stalls search
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Tree-structured Parzen Estimator (TPE) scale better than Gaussian Process-based Bayesian Optimization?
  • ?
    How does the Successive Halving Algorithm (ASHA) prune unpromising hyperparameter configurations early?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-086: Common Evaluation Metrics for Classification, Regression, and Ranking📋 Back to BankNext →M2-088: Hyperparameter Importance Hierarchy and Efficient Tuning Strategies