A complete tabular project, end to end
By the end: You have taken a real public dataset from raw table to a validated, saved model with error analysis, using the standard scikit-learn workflow.
- Loading and looking
- A baseline in one line
- Preprocessing in a pipeline
- Linear model, then gradient boosting
- Cross-validation and a held-out test
- Error analysis and saving the model
New to supervised learning? Collect ten thousand past house sales. For each one you have the details — size, postcode, number of rooms — and the price it actually sold for. That price is the label: the answer you already know. Supervised learning is handing all of those examples to an algorithm and letting it work out which combination of details predicts the price, so it can put a number on a house that has not sold yet. Why it matters
Most machine learning at most companies is still a table of rows and a column you want to predict. This section works one through end to end on a real public dataset: loading it and looking at it properly, getting a crude baseline in place before anything clever, building the preprocessing into a pipeline so it cannot leak, comparing a linear model against gradient boosting, then examining the errors rather than only the score. It is the workflow every scikit-learn project reuses.
Step Why it is in this order Skipping it costs you
1 look at the data you cannot choose a metric without it a metric nobody wanted
2 hold out a test set everything after this can overfit a number you cannot trust
3 dummy baseline gives every later number a meaning shipping 0.87 that was free
4 pipeline preprocessing refits per fold a score that will not reproduce
5 linear model tells you whether the problem is hard no reference point
6 boosting the actual candidate nothing, if 5 was good enough
7 test once the only unbiased estimate you get a slowly rotting test set
8 error analysis says what to do next a model you cannot improveThe data#
from sklearn.datasets import fetch_openml
adult = fetch_openml("adult", version=2, as_frame=True)
X, y = adult.data, (adult.target == ">50K").astype(int)
print(X.shape, round(y.mean(), 4)) # (48842, 14) 0.2393
print((X.dtypes == "category").sum()) # 8 categorical, 6 numeric
print(X.isna().sum()[X.isna().sum() > 0].to_dict())
# {'workclass': 2799, 'occupation': 2809, 'native-country': 857}Three facts from those four lines decide the rest of the project. The positive class is 24 per cent, so accuracy will be misleading and the metric should be ROC AUC or average precision rather than accuracy. Eight of the fourteen columns are categorical, which means either one-hot encoding for a linear model or a booster that accepts categories directly. And three columns have missing values, which is not a data error but an artefact of how the census was collected; an imputer plus an indicator will do, and boosting can take them as they are.
Split once, and hold out a test set#
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=0)This happens before anything is inspected in detail, and stratify keeps the 24 per cent positive rate identical in both halves. The test set is now closed. Every decision from here is made on cross-validation over the training set.
The baseline#
from sklearn.dummy import DummyClassifier
from sklearn.model_selection import cross_val_score
print(cross_val_score(DummyClassifier(), X_train, y_train, scoring="roc_auc").mean()) # 0.5A dummy scores exactly 0.5 ROC AUC by construction, which makes it look like a pointless step. It is not: the same dummy scores 0.76 on accuracy, and that is the number a stakeholder will hear first. Knowing that 0.76 is free is what stops anyone celebrating a model that scores 0.80.
A pipeline#
from sklearn.compose import make_column_transformer, make_column_selector
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
prep = make_column_transformer(
(make_pipeline(SimpleImputer(strategy="median"), StandardScaler()),
make_column_selector(dtype_include="number")),
(make_pipeline(SimpleImputer(strategy="most_frequent"), OneHotEncoder(handle_unknown="ignore")),
make_column_selector(dtype_include="category")),
)
logreg = make_pipeline(prep, LogisticRegression(max_iter=2000))
print(cross_val_score(logreg, X_train, y_train, cv=5, scoring="roc_auc").mean()) # 0.9055Everything that learns from data, the imputer, the scaler and the encoder, lives inside the pipeline, so cross-validation fits it on each training fold only. That one habit prevents the most common leakage in tabular work.
It is worth seeing how large that effect can be, because described in words it sounds like pedantry. Here is a dataset of pure noise with labels assigned at random, where the honest score is 50 per cent and nothing can beat it.
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
rng = np.random.default_rng(0)
Xn = rng.normal(0, 1, (200, 5000)) # pure noise
yn = rng.integers(0, 2, 200) # labels unrelated to it
# WRONG: pick the best 20 columns using every row, then cross-validate
best = SelectKBest(f_classif, k=20).fit(Xn, yn).transform(Xn)
print(cross_val_score(LogisticRegression(max_iter=2000), best, yn, cv=5).mean()) # 0.795
# RIGHT: selection inside the pipeline, refitted on each training fold
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=2000))
print(cross_val_score(pipe, Xn, yn, cv=5).mean()) # 0.510With 5,000 noise columns and 200 rows, some columns correlate with the labels by luck. Choosing them while looking at all the rows lets that luck into every fold, and the model then re-finds it. The same mechanism operates more quietly with a scaler or an imputer, where it inflates a score by a point or two rather than thirty, and is correspondingly harder to notice.
Gradient boosting#
from sklearn.ensemble import HistGradientBoostingClassifier
hgb = HistGradientBoostingClassifier(categorical_features="from_dtype", random_state=0)
print(cross_val_score(hgb, X_train, y_train, cv=5, scoring="roc_auc").mean()) # 0.9282That gap, a little over two points of AUC, is the honest size of the improvement you should expect from boosting on a table like this one. It is real and worth having, and it is not the order-of-magnitude difference the reputation suggests. If the linear model had come within half a point, the right decision would have been to ship the linear model.
Test once, then look at the errors#
from sklearn.metrics import roc_auc_score, classification_report
hgb.fit(X_train, y_train)
proba = hgb.predict_proba(X_test)[:, 1]
print(roc_auc_score(y_test, proba)) # 0.9283
print(classification_report(y_test, (proba > 0.5).astype(int), digits=3))
# precision recall f1-score support
# 0 0.895 0.943 0.918 7431
# 1 0.781 0.649 0.709 2338
# accuracy 0.873 9769The test AUC of 0.9283 against a cross-validated 0.9282 is the result you want: cross-validation was not optimistic, so the process was sound. The classification report is where the useful detail is. Recall on the positive class is 0.649, meaning the model misses roughly a third of the high earners, while its accuracy reads 0.873. If the application were finding people to contact, that 0.649 is the number that matters and it would be improved by lowering the threshold from 0.5, not by improving the model.
errors = X_test.assign(y=y_test, p=proba).query("(p > 0.5) != (y == 1)")
total = X_test.groupby("education", observed=True).size()
wrong = errors.groupby("education", observed=True).size()
print(((wrong / total).dropna().sort_values(ascending=False) * 100).round(1).head(6))
# Doctorate 18.8 overall error rate: 12.7
# Bachelors 17.7
# Masters 17.7
# Assoc-voc 16.6
# Prof-school 14.4
# Assoc-acdm 14.0The model is half again as wrong on doctorates as it is overall. That is the finding, and it is actionable in a way the single number 0.9283 is not: highly educated people are the group where education alone stops predicting income, so the model needs a feature that separates them, or the documentation needs to say the predictions are less reliable there. Reporting the AUC and stopping would have hidden this entirely.
The test set is used once, at the end. Then read the errors grouped by slice: the groups where the model fails most are where more data, better features, or a note in the model's documentation are needed.
Save it#
import joblib
joblib.dump(hgb, "income_model.joblib") # load with the same library versionsA pickle is not a model format
joblib writes a Python pickle, which stores references to scikit-learn's classes rather than the maths. Load it under a different scikit-learn version and you get either a warning, a silently different model, or an exception, and loading an untrusted one executes arbitrary code. Pin the library versions next to the file, and for anything crossing a team boundary export to ONNX instead.
Where people go wrong
Checking the test score after every change. After ten looks, the test set has become a validation set and the final number is optimistic.
If this comes up in an interview
Walk me through how you would approach a new tabular classification problem.
Look at the data and target balance, hold out a test set, get a dummy baseline, build a preprocessing pipeline so cross-validation is leak-free, try a regularised linear model then gradient boosting, choose on cross-validation, evaluate once on the test set, then analyse errors by slice and check calibration if probabilities are used.
Then they push further
Why do gradient boosted trees usually beat neural networks on tables?
Tabular features are heterogeneous, often with sharp thresholds and irregular interactions, which trees capture naturally. Trees need no scaling, handle missing values and categories well, and are strong with modest data. Neural networks shine when there is structure like images or text to exploit.
How would you explain a prediction to a stakeholder?
Show the main features driving the model overall with permutation importance, and for a single case, local explanations such as SHAP values, with the caveat that they explain the model's behaviour, not the real-world cause.
What gets marked up. Strong answers mention a baseline, a pipeline to avoid leakage, a single use of the test set, and slice-level error analysis.
Build this: Complete this project on the Adult dataset or another public table, then write a one-page model card: data, metric, performance overall and on three slices, known failure modes and intended use.
Watch it explained
This is one section of Supervised learning, 20 hours across 4 sections. Read it in full to get the whole topic in order.