The questions every ML interview asks, answered properly

Bias and variance, regularisation, missing data, why high-dimensional space behaves strangely: a small set of questions about classical machine learning comes up again and again, and the usual answers are memorised phrases that collapse under one follow-up. This section works through each of them properly, so what you leave with is an explanation you could defend rather than a definition you can recite.

How to use this section#

These questions come up in almost every ML interview loop. Turn on 'Test yourself' below, answer each out loud before revealing, and practise the follow-ups. The follow-ups are where interviews are actually decided.

First, though, the five ideas the questions are drawn from, each worked rather than defined. A memorised definition of the bias-variance trade-off survives one question. Having watched the two columns of numbers behave differently survives three.

Bias and variance, in two columns#

Bias is error from a model that cannot represent the pattern. Variance is error from a model that is too sensitive to the particular rows it was shown. You do not need a decomposition to tell them apart: fit the model, print the training error next to the validation error, and read them.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

rng = np.random.default_rng(0)
X = rng.uniform(-1, 1, (60, 1))
y = np.sin(3 * X[:, 0]) + rng.normal(0, 0.15, 60)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=0)

for deg in (1, 3, 9, 20):
    m = make_pipeline(PolynomialFeatures(deg), LinearRegression()).fit(Xtr, ytr)
    print(deg, round(np.mean((ytr - m.predict(Xtr)) ** 2), 4),
               round(np.mean((yte - m.predict(Xte)) ** 2), 4))

# degree  1   train 0.1694   test  0.3032    both bad        -> bias
# degree  3   train 0.0224   test  0.0317    close together  -> about right
# degree  9   train 0.0118   test  0.0485    gap opening     -> variance
# degree 20   train 0.0088   test 32.9775    best train, catastrophic test

training error   validation error   diagnosis   what actually helps
high             high, similar      bias        more features, more capacity, less regularisation
low              much higher        variance    more data, more regularisation, less capacity
low              low                neither     you are done; stop tuning
high             lower than train   a bug       leakage, a broken split, or mismatched sets

The follow-up that catches people is whether more data helps. It helps variance and does nothing for bias, and the difference is stark enough to measure in four lines.

for deg in (1, 15):
    for n in (30, 100, 1000, 10000):
        Xb = rng.uniform(-1, 1, (n, 1))
        yb = np.sin(3 * Xb[:, 0]) + rng.normal(0, 0.15, n)
        m = make_pipeline(PolynomialFeatures(deg), LinearRegression()).fit(Xb, yb)
        Xv = rng.uniform(-1, 1, (3000, 1))
        yv = np.sin(3 * Xv[:, 0]) + rng.normal(0, 0.15, 3000)
        print(deg, n, round(np.mean((yv - m.predict(Xv)) ** 2), 4))

# degree 1  (bias)      30: 0.1876   100: 0.1927   1000: 0.1877   10000: 0.1908
# degree 15 (variance)  30: 2559.14  100: 0.0307   1000: 0.0225   10000: 0.0223

Regularisation#

Regularisation adds a cost for complexity, so the model prefers a simpler explanation unless the data pays for the extra. L2 penalises squared weights and shrinks them smoothly; L1 penalises absolute weights and sets many exactly to zero, which makes it a feature selector. The strength is a hyperparameter and it is chosen by cross-validation, never by inspection.

The useful observation for an interview is that almost everything people do to stop overfitting is the same idea wearing different clothes. Early stopping limits how far the weights can travel from their small initial values, which is a budget on complexity. Dropout forces the network not to depend on any one unit. Data augmentation says the label should not change under transformations you specify. Max depth on a tree, and the number of boosting rounds, are the same lever again. If asked to name regularisation techniques, naming the mechanism they share is what separates a strong answer.

Choosing a model family#

This question is testing whether you have constraints in mind other than accuracy. Accuracy is where beginners stop, and it is rarely the binding constraint in a real deployment.

Constraint that actually binds        Pushes you towards
someone must sign off on decisions    logistic regression, a shallow tree
predictions in under a millisecond    linear model, or a small compiled tree ensemble
a few hundred rows                    regularised linear model
retraining daily on fresh data        anything that fits in minutes
features arrive with gaps             gradient boosting, which handles them natively
the answer must be a real probability logistic regression, or calibrate afterwards
accuracy above everything else        gradient boosting, tuned

Missing data#

The first question is never which imputation strategy. It is why the value is missing, because that determines whether imputing is honest. If a sensor fails at random, filling with the median loses almost nothing. If income is missing because the respondent declined to answer, and declining correlates with income, then the fact of missingness is one of the better features in the dataset and imputing it away destroys signal.

from sklearn.impute import SimpleImputer

imp = SimpleImputer(strategy="median", add_indicator=True)   # keeps the missingness

Two rules survive every variation of this question. Fit the imputer inside the cross-validation pipeline, or the median has been computed from rows the model is about to be tested on. And check whether the missingness pattern is the same in training and in production, because a column that is 2 per cent missing historically and 90 per cent missing in the live feed will not announce itself; the model will simply get quietly worse.

High dimensions#

Volume grows exponentially with dimensions, so data that felt dense in two dimensions is isolated points in two hundred. Covering the space to the same density needs exponentially more rows, which nobody has. The practical consequence is that distances stop discriminating: in a thousand dimensions the nearest of five hundred random points sits about 83 per cent as far away as the farthest, so any method built on neighbourhoods degrades towards a coin flip.

What follows is the answer to the follow-up about why anything works at all in high dimensions. Real data is not spread uniformly through its space; it lies near a much lower-dimensional surface within it, because the features are related. Feature selection, regularisation, PCA and embedding models are all ways of finding that surface and working on it instead. That is also the honest account of why vector search functions in 1,536 dimensions when the geometry above says it should not.