Hyperparameter tuning interview questions
5 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Tuning, in order of what matters
Read the section that teaches thisIf this comes up in an interview
Which hyperparameter would you tune first?
Learning rate. It has more effect than the rest combined. Search it by orders of magnitude first, then refine, and only then move to schedule, batch size and regularisation.
Why does random search beat grid search?
Most hyperparameters barely matter. A grid spends its budget evenly, so it tries only a few distinct values of the one that does. Random search tries a different value of every parameter on every run, exploring the important dimension far more thoroughly for the same compute.
Why do you need a validation set separate from test?
Because tuning decisions leak the score they are based on into the model. With enough trials you find a configuration suited to that specific set. Validation drives decisions; test is touched once for the reported number.
Learning rate schedules and a tuning workflow
Read the section that teaches thisIf this comes up in an interview
You have budget for 30 training runs. How do you tune?
One or two runs to confirm the baseline trains, a learning rate range test, then a random or Bayesian search over learning rate and regularisation on log scales with early pruning. Keep a few runs to rerun the top settings with different seeds, since the difference between the top two is often noise.
Then they push further
Why random search rather than a grid?
Usually only one or two hyperparameters matter. A grid wastes most runs repeating the same values of the important one, while random search tries a new value of every parameter on every run.
The best run beats the second best by 0.3%. Which do you pick?
Rerun both with several seeds first. If the gap is inside the seed-to-seed spread, choose on other grounds, such as the simpler or cheaper setting.
What gets marked up. Marked up for prioritising the learning rate, using log scales, pruning, and checking seed variance.
Why use learning rate warmup?
Early in training the weights and the optimiser's running statistics are poorly set, so a full learning rate can cause divergence. Ramping up over the first steps avoids that, especially for large models and large batches.
These are the questions for one topic. The whole question bank covers every topic on the site.