Hyperparameter Tuning
Stop blindly using sklearn defaults. The 1-hour workflow in this lesson takes most models from 'OK' to 'within 5% of the best you'll ever achieve'. The pros aren't smarter — they just have a better playbook.
Learning Objectives
After this lesson, you will be able to:
- Pick the right search strategy for your problem (grid search, random search, Bayesian optimization, or Hyperband) based on the size of the search space and your compute budget
- Define search spaces correctly using log-uniform vs uniform distributions, and avoid the most common silent mistake (uniform-tuning a learning rate)
- Use Optuna with TPE to tune real models and prune unpromising trials early, getting better results in a fraction of the compute of grid search
- Tune inside a cross-validation loop without leaking information into the test set, and know when to use nested CV for unbiased reporting
Prerequisites
Before You Start
Don't worry if the math behind Bayesian optimization feels intimidating at first — you can ship great results with
optuna.create_study() and three lines of code, and pick up the theory once you have the muscle memory.#War Story: The Hour That Replaced a Month of "Trying Things"
A retail analytics team had been hand-tuning a LightGBM demand forecaster for four weeks. They were stuck at RMSE 14.2. The lead claimed they had "tried everything" — different
num_leaves, different learning_rate, different min_data_in_leaf. Each round was an engineer typing values into Slack and waiting for results.The fix was a single Optuna script run in 62 minutes on the same dataset:
pythonrunnable cell
import optuna
def objective(trial):
params = {
'learning_rate': trial.suggest_float('lr', 1e-3, 0.3, log=True),
'num_leaves': trial.suggest_int('leaves', 16, 256),
'min_data_in_leaf': trial.suggest_int('min_data', 5, 100),
'feature_fraction': trial.suggest_float('ff', 0.5, 1.0),
'reg_alpha': trial.suggest_float('alpha', 1e-8, 1.0, log=True),
}
model = LGBMRegressor(**params, n_estimators=5000)
return cross_val_score(model, X, y, cv=5,
scoring='neg_root_mean_squared_error').mean()
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)Result: RMSE 11.8 — a 17% improvement the team thought was impossible. The TPE sampler had discovered a
learning_rate of 0.012 (the team's grid had been [0.01, 0.05, 0.1], and they had read on a blog that 0.05 was "the best default"). The blog was right on average, wrong for this dataset.Stop blindly using sklearn defaults. The 1-hour workflow below gets you to ~90% of expert performance on almost any tabular problem.
#The Real Job of an ML Engineer
A hyperparameter is a configuration knob you set before training —
learning_rate, max_depth, n_estimators, C, gamma. The model's parameters (weights, splits, support vectors) are what training learns. The hyperparameters are what you have to learn separately, by trying combinations and watching the validation score move.The naive approach — try a few values you read on Stack Overflow — is what most people do. It is also why most people leave 10–30% of their model's potential performance on the floor.
#The Four Strategies
#1. Grid Search: The One You Should Almost Never Use
Enumerate every combination of values from a fixed list per hyperparameter and evaluate all of them.
Use grid only when: you have ≤ 3 hyperparameters and want a clean, reproducible sweep for a paper or a regression test. For real tuning work, skip it.
#2. Random Search: The Default Baseline
Sample each hyperparameter independently from a distribution and run a fixed number of trials.
Bergstra and Bengio's central observation: in real ML problems, only a handful of hyperparameters drive most of the variance. Grid search wastes the rest of its budget on the irrelevant dimensions. Random search spends those trials on more values of the dimensions that matter.
pythonrunnable cell
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, randint
space = {
"learning_rate": loguniform(1e-3, 1e-0), # log-uniform: critical for LR
"max_depth": randint(3, 12),
"subsample": loguniform(0.5, 1.0),
}
search = RandomizedSearchCV(model, space, n_iter=100, cv=5)
search.fit(X_train, y_train)Use random when: you have a budget under a few hundred trials, your hyperparameters are mostly continuous, and you want a strong, simple baseline. Random search is what you should benchmark every fancier method against.
The interactive below makes the grid-vs-random difference visceral. Same model, same 2D hyperparameter space — flip strategies and watch how the dots fall on the validation-score landscape. The convergence strip on the right tracks how fast the best score is found.
Grid vs random vs Bayesian, where the budget actually landsInteractive
Loading visualization...
Quick check
In the visualization above, you set the budget to 16 trials. Grid search uses a 4x4 lattice and lands on the optimum. Why does random search often beat grid at the same budget on real problems even though grid 'covers' the space more uniformly?
#A concrete 4-grid-point example
To make grid search vs. random concrete: imagine you grid-search over
learning_rate ∈ {0.01, 0.1} and max_depth ∈ {4, 8}. Four cross-validated trials:| Trial | LR | Depth | CV AUC |
|---|---|---|---|
| 1 | 0.01 | 4 | 0.814 |
| 2 | 0.01 | 8 | 0.842 |
| 3 | 0.10 | 4 | 0.823 |
| 4 | 0.10 | 8 | 0.798 |
You'd pick trial 2 (LR=0.01, depth=8 → AUC 0.842). But the true optimum is probably at
(LR=0.03, depth=6) — somewhere between your grid points. Grid search can never see it. Random search with the same 4 trials samples 4 unique LRs and 4 unique depths and has a much higher chance of stumbling close to the optimum. Bayesian search uses results from trial 2 ("depth=8 helped, LR=0.01 helped") to bias trial 5 toward similar regions, and converges faster still.#The 1-Hour Tuning Workflow (the actually-useful checklist)
This is the workflow that gets you 90% of expert performance on almost any tabular problem:
- Split data into train / test, lock the test set away.
- Pick a baseline model. If unsure, gradient boosting (LightGBM or XGBoost) is almost always the right starting point for tabular.
- Wrap preprocessing in a
Pipelineso it fits per-fold (scaler, encoder, imputer all live in the pipeline, never on the full training set). - Define a log-uniform search space for rate-like hyperparameters (LR, alpha, gamma, dropout). Use
trial.suggest_float(..., log=True)in Optuna. - Run 50-100 Optuna trials with
MedianPrunerand 5-fold CV. Set a 60-minute wall-clock timeout. - Evaluate on the test set once. Report this single number as your honest generalization estimate.
That's it. Six steps, one hour, results that beat hand-tuning by 10-30% on typical problems.
#3. Bayesian Optimization: The One You Should Actually Use
The two dominant Bayesian methods in practice:
- Gaussian Process (GP): model the score as a function with a smooth GP prior. Best for small (≤ 20) continuous spaces. Used by
scikit-optimize,BoTorch. - Tree-structured Parzen Estimator (TPE): model the good trials and bad trials as two separate density estimates and sample from where the good density is high relative to the bad density. Handles categorical and conditional hyperparameters cleanly. Used by
Optuna,Hyperopt.
Switch the strategy to Bayesian (TPE) in the same visualization and watch what changes. After 5-10 warmup random samples, the next picks cluster near the best-so-far peak — TPE is exploiting the structure of past trials. Compare the convergence strip: Bayesian typically reaches a high score with fewer trials than random.
Now flip to Bayesian (TPE) and watch the search bias toward the high-score regionInteractive
Loading visualization...
Quick check
You're tuning 7 hyperparameters on a model where each CV fit takes 3 minutes. You can afford a wall-clock budget of about 4 hours. Which strategy gives the best expected validation score?
#4. Hyperband and Successive Halving: Compute-Aware Tuning
Use Hyperband when: training time per config is the bottleneck (deep learning, large GBMs with
n_estimators, multi-fold CV). For sub-minute trials, classical Bayesian without pruning is usually simpler.What Do You Think?
You have a 7-dimensional search space (5 continuous, 2 categorical), a budget of 100 trials, and each trial takes 2 minutes. Which strategy gives you the best expected validation score?
For 7 dimensions and 100 trials, Bayesian optimization (TPE) is the right call. Grid would be capped at 128 trials only if every dim had 2 values, which is far too coarse for continuous hyperparameters. Random gives you good coverage but spends every trial blind. TPE uses each result to inform the next — typical lift over random in this regime is 5–15% on validation score.
#How to Define Search Spaces (the silent killer)
This is where most tuning attempts die quietly.
#The hyperparameter cheat sheet (what actually matters per algorithm)
| Algorithm | Hyperparameters that matter | Distribution |
|---|---|---|
| Linear / Logistic regression | C (or alpha), l1_ratio | log-uniform |
| KNN | n_neighbors, weights, metric | int / cat |
| SVM (RBF) | C, gamma | log-uniform |
| Decision tree | max_depth, min_samples_leaf, min_samples_split | int |
| Random forest | n_estimators, max_depth, max_features, min_samples_leaf | int / fractional |
| Gradient boosting (XGB / LGBM / CB) | learning_rate, n_estimators (with early stopping), max_depth / num_leaves, min_child_weight / min_data_in_leaf, reg_alpha, reg_lambda, subsample, colsample_bytree | log-uniform / int |
For gradient boosters,
n_estimators should usually be set very large (e.g. 5000) and pinned by early stopping on a validation fold rather than tuned directly. Tune learning_rate instead and let early stopping pick the right number of trees for that LR.#Tuning Inside Cross-Validation
The cardinal rule: the validation set must not be touched during preprocessing. Standardization, target encoding, feature selection, missing-value imputation — anything that learns from data — has to be re-fit on the training portion of every CV fold, not on the full training set before splitting.
Interactive Lab
Step through K-Fold, Stratified, and TimeSeriesSplit fold-by-fold. The wrong splitter inside a tuning loop is the most common silent leak in production ML — this animation makes the difference obvious.
The clean way is
Pipeline:pythonrunnable cell
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform
pipe = Pipeline([
("scaler", StandardScaler()),
("clf", LogisticRegression(max_iter=2000)),
])
space = {"clf__C": loguniform(1e-4, 1e2), "clf__l1_ratio": [0.0, 0.5, 1.0]}
search = RandomizedSearchCV(pipe, space, n_iter=80, cv=5, scoring="roc_auc")
search.fit(X_train, y_train)Now
StandardScaler is fit inside each CV fold, on training data only. No leakage.#When you need nested CV
If you want to report the validation score of your tuning procedure as a generalization estimate (e.g. for a paper, a model card, or a regulatory filing), one round of CV is biased — you tuned to maximize that exact CV score, so it overestimates real performance. Nested CV gives an unbiased estimate at compute cost ~ inner_folds × outer_folds.
pythonrunnable cell
from sklearn.model_selection import cross_val_score, KFold
outer = KFold(n_splits=5, shuffle=True, random_state=0)
unbiased_scores = cross_val_score(search, X_train, y_train, cv=outer, scoring="roc_auc")
print(f"Nested CV: {unbiased_scores.mean():.3f} ± {unbiased_scores.std():.3f}")For day-to-day work, single-level CV with a held-out test set is sufficient. Reach for nested CV when you have to defend the number.
Quick check
You are tuning a fraud-detection model on transactions from Jan-Aug, and you'll deploy in Sep. Which CV splitter should you wrap inside your Optuna objective?
Quick check
Multiple measurements come from the same patient. You're tuning a clinical risk score with stratified k-fold. What's likely going wrong?
Run the next playground to see the gap between grid, random, and a hand-rolled Bayesian-style local search. Same model, same budget, three strategies — count how many evals each takes to land within 1% of the optimum.
Loading visualization...
#Putting It Together
pythonplayground.py · Pyodide
Tests · Verify random search returns a valid roc_auc score. Verify the test-set evaluation runs after tuning, not during.
#Pruning, Warm Starts, and Multi-Fidelity
A few production-grade tricks that compound:
- Pruning: With Optuna's
MedianPrunerorHyperbandPruner, a trial reports its intermediate score (e.g. score after 10/50/100 trees) and is killed if it falls below the median of past trials at the same step. Typical compute savings: 30–60%. - Warm starting: Save your
studyto a database (storage="sqlite:///study.db") and resume next week with prior trials informing TPE. This compounds across projects. - Multi-fidelity: Tune on a 10% subsample first to find a good neighborhood, then do a small refinement run on the full data with the narrowed search space.
Quick check
A reviewer asks 'what's your model's expected AUC on truly unseen data after the full tuning procedure?' You ran 100 Optuna trials with 5-fold CV and report the best CV AUC. Why is that number wrong?
Quick check
You set 'n_estimators': trial.suggest_int('n_est', 100, 2000) AND 'learning_rate': trial.suggest_float('lr', 0.001, 0.5, log=True) in the same Optuna study for XGBoost. What's the more idiomatic choice?
#Key Takeaways
- Random search beats grid search at the same budget. Bergstra & Bengio's 2012 paper proved this for any realistic problem; the only reason to grid-search is small reproducible sweeps for papers
- Bayesian optimization (TPE via Optuna) is the practical default. It uses every past trial to inform the next, typically delivering 5–15% lift over random at the same compute on multi-dimensional spaces
- Use log-uniform distributions for rate-like hyperparameters. Learning rate, C, alpha, gamma, dropout — sampling uniformly is the most common silent bug in tuning, often costing more than picking the wrong algorithm
- Tune inside cross-validation, never on the test set. Wrap preprocessing in a
Pipelineso it fits per-fold; reach for nested CV when the tuning procedure's score itself needs to be reported unbiased - Pruning compounds with Bayesian search. Hyperband / median pruning kills underperforming trials early, often 30–60% compute saving with no quality loss; combine with TPE for the strongest off-the-shelf setup
#Quick Check
Quick Check1 / 4
Why does random search outperform grid search in practice for the same number of trials?
#The Bridge Forward
Tuning sets the dial between underfitting and overfitting that you learned to read in Bias-Variance & Learning CurvesBias-Variance TradeoffThe bias-variance tradeoff describes how decreasing a model's bias (underfitting) typically increases its variance (overfitting), and vice versa.Learn more → — every hyperparameter is a knob on the U-curve. The penalty terms from RegularizationRegularizationRegularization adds a penalty term to the loss function (L1, L2) to discourage overly complex models and reduce overfitting.Learn more → (alpha, l1_ratio) are the most common things you tune. And the metric you maximize during tuning is the one you learned about in Model Evaluation — pick wrong (e.g. accuracy on imbalanced data) and you tune your way to a useless model.
The next lesson, Model Interpretation, is what you reach for once tuning is done: now that you have a strong model, what is it actually doing, and can you defend it to a human?
A tuned model is the difference between a prototype and a product. Next up: Model Interpretation & Explainability — once your tuned model works, how do you actually explain why it works to a human, a regulator, or yourself six months from now.