Hyperparameter search and Bayesian optimisation

In short: Hyperparameters are settings fixed before training (learning rate, tree depth, number of layers). Searching for good values costs a lot of compute, so there are clever methods.

Simple ways

• Grid search: try all combinations of a grid. Thorough, but the number explodes.
• Random search: random combinations. Often better than grid search because only a few hyperparameters really matter and random values test more different values of those important ones.

Bayesian optimisation

The idea: from the trials so far you build a surrogate model that estimates how good a new value will probably be and how uncertain this estimate is (often a Gaussian process or TPE). An acquisition function picks the next spot with high hope or high uncertainty. This needs far fewer trials.

Tools: Optuna, Hyperopt, Ray Tune.

Practical tips

1. Evaluate on a validation set, never on the test set (training, validation and test).
2. Stop bad trials early (successive halving, Hyperband).
3. Most important first: learning rate (batch, epoch and learning rate), then model size.
4. Fix random seeds so results can be reproduced.

See also: Optimisers: SGD, momentum and Adam