Grid vs Random vs Bayesian Search

The Hyperparameter Optimization framework (Grid vs Random vs Bayesian Optimization) establishes the mathematical foundations to maximize model performance under strict compute budgets; 3 primary paradigms: 1) Grid Search: Cartesian product evaluation scaling exponentially ($|G| = \prod m_i$) with dimension $D$, suffering curse of dimensionality; 2) Random Search: uniform/log-uniform sampling across continuous subspaces; mathematically proven by Bergstra & Bengio that $N=60$ independent random trials yield a 95% probability of finding a hyperparameter configuration within the top 5% true optimum, vastly outperforming Grid Search on high-dimensional spaces with low effective rank; 3) Bayesian Optimization (Gaussian Processes, TPE via Optuna): constructing surrogate posterior response surfaces and maximizing Acquisition Functions (Expected Improvement EI, Upper Confidence Bound UCB) to intelligently balance exploration against exploitation.

Deep learning architecture hyperparameter tuning, RL hyperparameter search, and Automated Machine Learning (AutoML).

Manual hyperparameter trial-and-error wastes thousands of GPU hours; statistical search algorithms guarantee optimal exploration under fixed compute budgets.

Derive the mathematical proof showing why $N=60$ random trials achieves $>95\%$ probability of catching the top 5% true optimum region?

Derive the Expected Improvement (EI) acquisition function formula under Gaussian Process posterior distributions in Bayesian optimization?

Why must scale hyperparameters (learning rates $\eta$, weight decay $\lambda$) be sampled log-uniformly rather than linearly?

Explain how Successive Halving and Hyperband (ASHA) dynamically allocate compute by aggressively pruning unpromising configurations early?

Why is reporting total search spaces and tuning budgets mandatory to avoid over-tuning optimism bias in academic literature?