What is Hyperparameter Tuning?
Model parameters are learned during training (weights, tree splits). Hyperparameters are set before training — they control the learning process itself: number of trees, learning rate, regularisation strength, kernel type. Tuning finds the hyperparameter combination that maximises validation performance.
Always tune on validation data, evaluate on test data. If you use test set performance to guide hyperparameter choices, you've leaked test information into your model and your results are optimistically biased.
GridSearchCV — Exhaustive Search
Tries every combination of the specified hyperparameter grid. Guaranteed to find the best combination within the grid — but cost grows exponentially with the number of parameters and values.
Python
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import pandas as pd
X, y = load_breast_cancer(return_X_y=True)
# Grid search on Random Forest
param_grid = {
"n_estimators": [50, 100, 200],
"max_depth": [None, 5, 10],
"min_samples_leaf": [1, 2, 5],
"max_features": ["sqrt", "log2"],
}
# 3 × 3 × 3 × 2 = 54 combinations × 5 folds = 270 fits
gs = GridSearchCV(
RandomForestClassifier(random_state=42),
param_grid,
cv=5, scoring="roc_auc",
n_jobs=-1, verbose=1,
return_train_score=True
)
gs.fit(X, y)
print(f"Best AUC: {gs.best_score_:.4f}")
print(f"Best params: {gs.best_params_}")
# Inspect full results as a DataFrame
results_df = pd.DataFrame(gs.cv_results_)
results_df = results_df.sort_values("mean_test_score", ascending=False)
print(results_df[["param_n_estimators","param_max_depth","mean_test_score","std_test_score"]].head(10))RandomizedSearchCV — Efficient Search
Samples a fixed number of hyperparameter combinations at random from specified distributions. Empirically, random search finds equally good results as grid search in 1/10 the time — because most hyperparameters have low importance and random search avoids redundant combinations.
Python
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform
import xgboost as xgb
# Define distributions (not discrete lists)
param_dist = {
"n_estimators": randint(100, 800), # uniform integer in [100, 800)
"max_depth": randint(3, 10),
"learning_rate": uniform(0.01, 0.29), # uniform float in [0.01, 0.30)
"subsample": uniform(0.5, 0.5), # [0.5, 1.0)
"colsample_bytree": uniform(0.5, 0.5),
"reg_alpha": uniform(0.0, 1.0),
"reg_lambda": uniform(0.5, 2.0),
"min_child_weight": randint(1, 10),
}
rs = RandomizedSearchCV(
xgb.XGBClassifier(eval_metric="logloss", random_state=42),
param_dist,
n_iter=50, # try 50 random combinations (vs. thousands for full grid)
cv=5, scoring="roc_auc",
n_jobs=-1, verbose=1, random_state=42
)
rs.fit(X, y)
print(f"Best AUC: {rs.best_score_:.4f}")
print(f"Best params: {rs.best_params_}")Bayesian Optimisation
The smartest search strategy. It builds a probabilistic surrogate model of the objective function (typically a Gaussian Process or Tree-structured Parzen Estimator), then uses an acquisition function to decide where to sample next — balancing exploration of unknown regions with exploitation of known good ones.
OptunaModern, fast, framework-agnostic. Uses Tree-structured Parzen Estimator (TPE). Best general-purpose choice.
scikit-optimize (skopt)Gaussian Process-based. Integrates cleanly with scikit-learn API. Good for smaller search spaces.
HyperoptTPE-based, supports conditional search spaces. Popular in production pipelines.
Python
import optuna
from sklearn.model_selection import cross_val_score
import xgboost as xgb
optuna.logging.set_verbosity(optuna.logging.WARNING)
def objective(trial):
params = {
"n_estimators": trial.suggest_int("n_estimators", 100, 600),
"max_depth": trial.suggest_int("max_depth", 3, 10),
"learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
"subsample": trial.suggest_float("subsample", 0.5, 1.0),
"colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 1.0),
"reg_alpha": trial.suggest_float("reg_alpha", 1e-3, 1.0, log=True),
"reg_lambda": trial.suggest_float("reg_lambda", 0.5, 3.0),
}
model = xgb.XGBClassifier(**params, eval_metric="logloss", random_state=42)
return cross_val_score(model, X, y, cv=5, scoring="roc_auc").mean()
study = optuna.create_study(direction="maximize",
sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=60, show_progress_bar=True)
print(f"
Best AUC: {study.best_value:.4f}")
print(f"Best trial: {study.best_params}")
# Plot optimisation history
optuna.visualization.plot_optimization_history(study).show()
optuna.visualization.plot_param_importances(study).show()Search Strategy Comparison
GridSearchCVExhaustive. Best when few parameters and discrete values. Cost: $O(\prod |\text{values}|)$. Use for final fine-tuning around a known good region.
RandomizedSearchCVRandomly samples $n$ combinations. Much more efficient. Great default for initial broad search. Use continuous distributions, not lists.
Bayesian (Optuna)Learns from previous trials. Converges to good regions faster than random. Best for expensive evaluations or large search spaces. The modern standard.
Practical workflow: (1) RandomSearch with 50 trials to find good regions. (2) Bayesian Optimisation with 100 trials around those regions. (3) GridSearch to fine-tune the top 2–3 parameters. Each stage feeds the next.
Summary
- GridSearch: exhaustive, guaranteed optimal within grid. Exponential cost.
- RandomSearch: samples $n$ combinations. Finds similarly good results 10× faster. Use distributions, not lists.
- Bayesian (Optuna): builds a surrogate model; uses past trials to guide next sample. Best for large/expensive searches.
- Always tune on validation/CV data. Never let test set results guide tuning decisions.