M2-088M2: Classical Machine LearningEvaluation Metrics & Hyperparameter TuningMedium
⚡Flashcard Drill➔Mastery:

Hyperparameter Importance Hierarchy and Efficient Tuning Strategies

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-089 →
📐 Mathematical Definition
importancei=Varhi[E[perf∣hi]]\text{importance}_i=\mathrm{Var}_{h_i}\big[\mathbb E[\text{perf}\mid h_i]\big]
⚡ One-Sentence Core Answer
Prioritize learning rate and regularization first, architecture second, and secondary parameters last; employ coarse-to-fine search and early-pruning strategies.

📌 Key Takeaways

  • •
    Tier 1: Learning rate, optimizer type, batch size (determine convergence dynamics)
  • •
    Tier 2: Regularization (weight decay, dropout, tree max depth, min child weight)
  • •
    Tier 3: Secondary architecture parameters (kernel sizes, activation functions, warmup schedules)
  • •
    Strategy: Coarse-to-fine random search combined with early-stopping heuristics (ASHA/Hyperband)

📐 Mathematical Derivations

Sensitivity Analysis: The response surface of objective metric y=f(θ1,…,θd)y = f(\theta_1, \dots, \theta_d) exhibits highly unequal variance across parameters. Using functional ANOVA (fANOVA), total variance is decomposed into marginal parameter contributions: Var(y)=∑iVi+∑i<jVij+…\text{Var}(y) = \sum_{i} V_i + \sum_{i < j} V_{ij} + \dots. Empirical benchmarks (e.g., Bergstra & Bengio) demonstrate that Learning Rate accounts for >50%>50\% of performance variation across optimization algorithms. Setting learning rate on a logarithmic scale (e.g., 10−510^{-5} to 10−110^{-1}) is mathematically mandatory because gradient descent step sizes scale multiplicatively.

🏭 Production Trade-offs

Efficient practical workflow: ① Stage 1 (Coarse Screening): Run 50–100 trials of Random Search or TPE across broad logarithmic bounds using aggressive early stopping (pruning after 20% of epochs). ② Stage 2 (Fine Tuning): Narrow bounds around the top 3 configurations and search with higher fidelity (more epochs, multiple random seeds). ③ Coupled Parameters: Tune jointly coupled parameters (e.g., learning rate and batch size, or tree depth and learning rate) rather than varying one at a time.
⚠️ Common Interview Pitfalls
  • ✕
    Tuning learning rate on a linear scale instead of a logarithmic scale
  • ✕
    Tuning low-impact secondary parameters (e.g., epsilon in Adam) while leaving learning rate unoptimized
🎯 Interviewer Follow-up Inquiries
  • ?
    Why must learning rate searches be conducted across logarithmic scales rather than uniform intervals?
  • ?
    How does the linear scaling rule relate batch size increases to learning rate adjustments in distributed SGD?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-087: Comparing Grid Search, Random Search, and Bayesian Optimization📋 Back to BankNext →M2-089: Handling Randomness and Seed Variance in Hyperparameter Tuning