Hyperparameter Tuning Beyond Grid Search

The one idea: Grid search spends its budget evenly across every hyperparameter, whether or not that hyperparameter actually matters — and in most real models, a couple of hyperparameters do almost all the work while the rest barely move the score. Random search spends the same budget unevenly, by chance, which sounds worse and is usually better: at equal cost it tests far more distinct values of the hyperparameters that matter, because it isn't wasting draws on a fine grid over the ones that don't.

⚠️ The trap: "We ran the full grid, so we know we found the best combination" is a comforting sentence and often a false one. A grid only ever tests the exact values you put on it — 100, 150, 200 trees, say — and if the true optimum sits at 130, the grid will never find it no matter how exhaustively it's searched, because 130 was never a candidate. Exhaustive search over the wrong candidates is not the same as finding the best hyperparameters; it's finding the best hyperparameters among the ones you happened to write down.

The Cross-Validation and Grid Search lesson covers how GridSearchCV works and why it's the honest way to tune: define a grid, run full k-fold CV on every combination, keep the winner, and never let the tuning process touch the final test set. All of that stays true here. What changes in this lesson is what goes inside the search — grid search is one strategy for walking the hyperparameter space, and past a small number of hyperparameters, it's rarely the best one.

Why grid search's exhaustiveness is a…