Learning a mapping from examples
By the end: You can train a model on tabular data and know which algorithm to reach for.
- Regression and classification
- Linear and logistic regression
- Trees and gradient boosting
- Why boosted trees still win on tables
New to supervised learning? Collect ten thousand past house sales. For each one you have the details — size, postcode, number of rooms — and the price it actually sold for. That price is the label: the answer you already know. Supervised learning is handing all of those examples to an algorithm and letting it work out which combination of details predicts the price, so it can put a number on a house that has not sold yet. Why it matters
This is the ordinary case that most applied machine learning still is: you have examples where the answer is known, and you want something that gives the answer for cases where it is not. The section covers the two shapes that problem takes, predicting a number and predicting a category, the handful of algorithms worth knowing for each, and why gradient-boosted trees remain the first thing to reach for on data that lives in a table.
What you have Reach for first
a table, predict a number gradient boosting regressor
a table, predict a category gradient boosting classifier
a table, and someone must sign off the model logistic or linear regression
under a few hundred rows regularised linear model
text, images, audio a pretrained network, fine-tuned
you need a number to beat before anything else predict the mean, or the majority classTwo problems#
Supervised learning takes two shapes. Regression predicts a number: a price, a duration, a temperature. Classification predicts a category: spam or not, which of five classes. The distinction decides your loss function and your metrics, and nearly everything follows from it.
Regression is scored by how far off the number is, usually mean squared error during training and mean absolute error when reporting to anyone who has to act on it. Classification is scored on whether the category was right, and trained on cross-entropy, which is the loss that punishes being confidently wrong. Picking the wrong pair is not a subtle error: a classifier trained with squared error learns to hedge towards the middle instead of committing, because squared error treats a confident mistake far too gently.
Linear regression#
Linear regression assumes the answer is a weighted sum of the features plus a constant, and finds the weights that minimise squared error. There is a closed-form solution, so there is no training loop and no learning rate; it fits in milliseconds on data that would take a network an hour.
The reason to reach for it is that every coefficient has a unit and a meaning. If size is measured in square metres and the answer in thousands of pounds, a coefficient of 3.1 says one more square metre is worth three thousand one hundred pounds, holding everything else fixed. That last clause is the whole of the difficulty, and the next block is what happens when it stops being true.
import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(0)
n = 2000
size = rng.normal(90, 25, n) # square metres
age = rng.integers(0, 60, n).astype(float) # years
price = 3.1 * size - 1.4 * age + 40 + rng.normal(0, 8, n) # thousands
lin = LinearRegression().fit(np.c_[size, age], price)
print(lin.coef_.round(2)) # [ 3.11 -1.38] -> recovers 3.1 and -1.4
print(round(lin.intercept_, 1)) # 39.0 -> recovers 40
print(round(lin.score(np.c_[size, age], price), 3)) # 0.991Now add a third feature that is the first one in different units. The model is just as accurate and the coefficients become nonsense.
size_sqft = size * 10.764 + rng.normal(0, 0.5, n) # the same thing, in square feet
Xc = np.c_[size, age, size_sqft]
for seed in (0, 1, 2):
idx = np.random.default_rng(seed).choice(n, n, replace=True) # resample the rows
m = LinearRegression().fit(Xc[idx], price[idx])
print(f"size {m.coef_[0]:+8.2f} size_sqft {m.coef_[2]:+7.2f} R2 {m.score(Xc, price):.3f}")
# size +11.37 size_sqft -0.77 R2 0.991
# size +5.49 size_sqft -0.22 R2 0.991
# size +11.14 size_sqft -0.75 R2 0.991Linear regression fails in four ways, in the order you will meet them. Correlated features, as above: predictions fine, coefficients meaningless, and you only notice if you refit. A genuinely curved relationship, which it fits with a straight line and is wrong at both ends. Outliers, because squared error weights a point ten times too far by a hundred. And extrapolation: it will confidently price a 400 square metre house from data that stops at 200, because nothing in the model knows the data ran out.
Logistic regression#
The same weighted sum, passed through a function that squashes any number onto the range zero to one so it can be read as a probability. The sum itself is the log-odds, and that is what makes the coefficients readable: a coefficient of minus 0.075 means one more unit multiplies the odds by exp(-0.075), which is 0.93, a seven per cent reduction in the odds per unit. Ten units multiply the odds by 0.47, roughly halving them.
from sklearn.linear_model import LogisticRegression
income = rng.normal(50, 15, n)
odds = np.exp(-0.08 * income + 3.0)
default = (rng.random(n) < odds / (1 + odds)).astype(int)
log = LogisticRegression().fit(income.reshape(-1, 1), default)
b = log.coef_[0, 0]
print(round(b, 4)) # -0.0748 recovers the -0.08 it was built from
print(round(np.exp(b), 4)) # 0.9279 odds multiplier per extra unit of income
print(round(np.exp(10 * b), 4)) # 0.4733 odds roughly halve per 10 unitsIt fails in the same four ways as linear regression, plus one of its own: the decision boundary is still a straight line through feature space, so any problem where the classes are separated by a curve needs the curve supplied as a feature. It is also the only model here that gives well-calibrated probabilities without extra work, which is why it survives in credit scoring and medicine long after it stopped being the most accurate option.
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
model.score(X_test, y_test)
model.coef_ # which features mattered, and in which directionAlways build the baseline
Before anything complicated, get a number from logistic regression or even from predicting the majority class. Without it you cannot tell whether your neural network is good or merely functional.
Decision trees#
A tree asks one yes-or-no question about one feature at a time, and each answer sends the example down a branch to another question. It handles curves, thresholds and interactions between features without being told they exist, needs no scaling, and can be read aloud as a set of rules.
A split is chosen by brute force. At each node the tree tries every feature and every threshold in the data, scores each candidate by how pure the two resulting groups are, and keeps the best. Purity for classification is usually the Gini impurity: the chance of being wrong if you guessed a label at random from the group's own distribution. A group that is all one class scores zero; an even split of two classes scores 0.5.
def gini(y):
if len(y) == 0:
return 0.0
p = np.bincount(y, minlength=2) / len(y)
return 1.0 - (p ** 2).sum()
def best_split(x, y):
parent = gini(y)
best_gain, best_t = 0.0, None
for t in np.unique(x)[:-1]: # every value is a candidate threshold
left, right = y[x <= t], y[x > t]
child = (len(left) * gini(left) + len(right) * gini(right)) / len(y)
if parent - child > best_gain:
best_gain, best_t = parent - child, t
return best_gain, best_t
x = np.array([22, 25, 29, 31, 35, 38, 42, 47, 51, 60])
y = np.array([ 0, 0, 0, 1, 0, 1, 1, 1, 1, 1])
gain, t = best_split(x, y)
print(round(gini(y), 3)) # 0.48 -> the parent node, five of each class
print(round(gain, 3), int(t)) # 0.32 35 -> split at age <= 35threshold left labels right labels weighted gini gain
age <= 25 0 0 0 1 0 1 1 1 1 1 0.300 0.180
age <= 29 0 0 0 1 0 1 1 1 1 1 0.171 0.309
age <= 35 0 0 0 1 0 1 1 1 1 1 0.160 0.320 <- chosen
age <= 38 0 0 0 1 0 1 1 1 1 1 0.267 0.213
age <= 47 0 0 0 1 0 1 1 1 1 1 0.400 0.080A tree left to run stops only when each leaf is pure, which on real data means one leaf per training row and a model that has memorised rather than learned. The controls are max_depth, min_samples_leaf and max_leaf_nodes, and they are the entire defence. A single tree constrained enough not to overfit is usually too weak to be interesting, which is why nobody ships one and why the next two sections exist.
Random forests#
Grow hundreds of deep trees, each on a bootstrap resample of the rows, and at each split let the tree consider only a random subset of the features. Average their predictions. Each tree individually overfits; the errors they make are different errors, and averaging cancels them.
The arithmetic of why is worth knowing, because it also says where the limit is. Averaging n predictions with variance v and correlation rho between them gives variance rho times v plus (1 minus rho) times v over n. The second term vanishes as you add trees. The first does not.
def ensemble_var(n_trees, rho, var=1.0):
return rho * var + (1 - rho) * var / n_trees
for n in (1, 10, 100, 1000):
print(n, round(ensemble_var(n, 0.0), 3), round(ensemble_var(n, 0.6), 3))
# 1 1.000 1.000
# 10 0.100 0.640
# 100 0.010 0.604
# 1000 0.001 0.600 <- adding trees past this point buys nothingSo the number of trees is not a parameter to tune, only a budget: more is never worse, and past a few hundred it stops helping. The parameter that matters is max_features, the size of that random subset, because it sets rho directly. Forests are hard to overfit, need almost no tuning, and land a little behind well-tuned boosting, which is a good trade when nobody has time to tune anything.
Gradient boosting#
Train a small tree, look at what it got wrong, train another tree on those errors, repeat. Each one corrects its predecessors. This is the strongest general approach for tabular data and remains so despite everything that has happened in deep learning.
The whole method fits in a dozen lines. Start from the best constant, which for squared error is the mean. At each round compute the residual, which for squared error is exactly the negative gradient of the loss. Fit a shallow tree to that residual, and add a small fraction of its prediction. That fraction is the learning rate.
from sklearn.tree import DecisionTreeRegressor
rng = np.random.default_rng(1)
X = rng.uniform(-3, 3, (400, 1))
y = np.sin(X[:, 0] * 2) + 0.3 * X[:, 0] ** 2 + rng.normal(0, 0.1, 400)
pred = np.full(len(y), y.mean()) # the best constant, before any tree
trees, lr = [], 0.1
for round_ in range(200):
residual = y - pred # for squared error, this IS the negative gradient
stump = DecisionTreeRegressor(max_depth=2).fit(X, residual)
pred += lr * stump.predict(X) # a small step in the direction it points
trees.append(stump)
if round_ in (0, 50, 100, 199):
print(round_, round(np.mean((y - pred) ** 2), 4))
# round 0 mse 1.0587
# round 50 mse 0.0186
# round 100 mse 0.0076
# round 199 mse 0.0058Because each step is small and deliberate, a boosted model with too many rounds overfits, unlike a forest. Early stopping on a validation split is the standard defence and is built in.
from sklearn.ensemble import HistGradientBoostingClassifier
model = HistGradientBoostingClassifier(
max_iter=300,
learning_rate=0.1,
early_stopping=True,
).fit(X_train, y_train)Hyperparameter What it does Where to start
learning_rate size of each correction 0.05 to 0.1
max_iter number of trees as high as you can afford
max_depth interaction order each tree can capture 3 to 6, or leave unset
min_samples_leaf refuses splits on too few rows 20, raise if overfitting
l2_regularization shrinks leaf values 0, raise if overfitting
max_features fraction of columns per split 1.0, lower on wide datafrom sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import train_test_split
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
for lr_, iters in [(1.0, 20), (0.3, 60), (0.1, 200), (0.03, 700)]:
m = HistGradientBoostingRegressor(learning_rate=lr_, max_iter=iters,
early_stopping=False, random_state=0).fit(Xtr, ytr)
print(lr_, iters, round(np.mean((yte - m.predict(Xte)) ** 2), 4))
# 1.0 20 trees test mse 0.0321
# 0.3 60 trees test mse 0.0293
# 0.1 200 trees test mse 0.0289 <- best
# 0.03 700 trees test mse 0.0291 <- 3.5x the compute, very slightly worseAs for which boosting library: HistGradientBoosting ships with scikit-learn and needs nothing installed, which is why it is used throughout here. LightGBM is usually the fastest on wide data, XGBoost is the one most job descriptions name, and CatBoost handles high-cardinality categorical columns with the least work. They are within a point or two of each other on most datasets; the choice rarely decides a project and never decides an interview.
An end-to-end fit#
The order matters more than any single model. Baseline first, so every later number means something; a linear model next, because it is fast and tells you whether the problem is even hard; then the ensembles. Everything compared by cross-validation on the training set, and the test set touched once.
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=0) # stratify keeps the class balance
models = {
"majority class": DummyClassifier(strategy="most_frequent"),
"logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),
"random forest": RandomForestClassifier(n_estimators=300, random_state=0),
"gradient boosting": HistGradientBoostingClassifier(random_state=0),
}
for name, m in models.items():
print(name, round(cross_val_score(m, X_train, y_train, cv=5, scoring="roc_auc").mean(), 4))
# majority class 0.5000
# logistic 0.9945
# random forest 0.9956
# gradient boosting 0.9923 <- boosting loses here, and that is the point of the tableBoosting coming third is not a mistake in the code. This dataset has thirty smooth, correlated measurements and 569 rows, which is close to the ideal case for a linear model and close to the worst case for trees. Reaching for boosting by reflex and reporting 0.992 would have left a simpler, faster, more explainable model that scores higher on the table, and nobody would ever have known.
logit = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)).fit(X_train, y_train)
coef = logit[-1].coef_[0]
for i in np.argsort(-np.abs(coef))[:3]:
print(f"{X.columns[i]:22} {coef[i]:+.2f} odds x{np.exp(coef[i]):.2f} per SD")
# radius error -1.32 odds x0.27 per SD
# worst radius -0.98 odds x0.37 per SD
# worst texture -0.97 odds x0.38 per SD
from sklearn.metrics import roc_auc_score
best = HistGradientBoostingClassifier(random_state=0).fit(X_train, y_train)
print(round(roc_auc_score(y_test, best.predict_proba(X_test)[:, 1]), 4)) # 0.9868Why trees beat neural networks on tables#
- Tabular features are heterogeneous and often uninformative individually. Trees select; networks must learn to ignore.
- Trees are invariant to monotonic transformations of a feature, so scaling and skew do not matter.
- They handle missing values natively rather than requiring imputation.
- Tabular datasets are usually small by deep learning standards, and networks need data.
Worth saying in an interview
Reaching for a neural network on a spreadsheet is a recognisable inexperience signal. Saying you would start with gradient boosting and only move on with evidence reads as having shipped something.
Where people go wrong
Skipping the baseline. Without one, a model scoring 0.87 is meaningless, because you do not know whether predicting the majority class would have scored 0.85.
If this comes up in an interview
You have a tabular dataset and a classification task. What do you try first?
A trivial baseline, then logistic regression for an interpretable reference, then gradient boosted trees, which are the strongest general option on tabular data. A neural network only with evidence that it beats boosting on this dataset.
Why do gradient boosted trees still beat neural networks on tabular data?
Tabular features are heterogeneous and many are uninformative, and trees select among them rather than having to learn to ignore. Trees are also invariant to feature scaling, handle missing values natively, and work on the smaller datasets typical of tabular problems.
What is the difference between bagging and boosting?
Bagging, as in random forests, trains many independent models on resampled data and averages them, which reduces variance. Boosting trains models sequentially with each correcting the previous one's errors, which reduces bias and usually performs better with tuning.
Build this: Take a tabular dataset and build four models in order: majority class, logistic regression, random forest, gradient boosting. Record all four scores and note how much each step actually bought.
Watch it explained
This is one section of Supervised learning, 20 hours across 4 sections. Read it in full to get the whole topic in order.