Feature engineering

A model cannot use a column that says "London" or a price that ranges from 3 to 300,000 alongside an age that ranges from 18 to 90. Feature engineering is the work of turning raw columns into numbers a model can actually learn from: encoding categories, putting different scales onto comparable footing, and building new columns that express something the raw ones only imply. The danger throughout is leakage — accidentally handing the model a column that quietly contains the answer.

What the column is                            Encode it as
under ~15 values, no order, linear model      one-hot
under ~15 values, no order, tree model        integer codes, or native categorical
a real order: small, medium, large            ordinal, in the right order
hundreds to thousands of values               target encoding, out of fold
thousands, and you cannot afford a lookup     hashing into a fixed width
you only need "how common is this"            frequency encoding
free text                                     TF-IDF, or sentence embeddings

Categories#

• One-hot: a column per category. Safe, and explodes with high cardinality.
• Ordinal: an integer per category. Only when the order is real, or you invent a ranking the model will believe.
• Target encoding: replace the category with the mean target for it. Powerful, and leaks badly unless computed inside the fold.
• Trees handle high-cardinality categories better than linear models do, which affects the choice.

One-hot is the default because it assumes nothing. Each category gets a column that is one when present and zero otherwise, so the model can learn an entirely separate effect for each with no ordering implied. What it costs is width. A column with ten thousand cities becomes ten thousand columns, almost all zero, which is fine for a linear model on a sparse matrix and genuinely bad for a tree: a tree has to find the one useful split among ten thousand near-empty columns, and each split isolates a single category rather than a group.

It also has a serving problem that only shows up in production. A category appears in live traffic that was not in the training data, and the encoder either raises or silently produces an all-zero row. Decide which you want, in writing: either map unknowns to a dedicated other column that existed at training time, or fail the request. Both are defensible; discovering the choice by reading a stack trace is not.

import numpy as np

def one_hot(values, categories):
    """Unknown values become an extra 'other' column, which must exist in training."""
    index = {c: i for i, c in enumerate(categories)}
    out = np.zeros((len(values), len(categories) + 1))
    for row, v in enumerate(values):
        out[row, index.get(v, len(categories))] = 1.0
    return out

cats = ["leeds", "bristol", "cardiff"]
print(one_hot(["bristol", "leeds", "york"], cats))
# [[0. 1. 0. 0.]
#  [1. 0. 0. 0.]
#  [0. 0. 0. 1.]]   <- york was never seen, and lands in 'other'

Ordinal encoding replaces each category with an integer, which is one column instead of ten thousand. The cost is that the integers mean something to some models. A linear model reads 0, 1, 2 as a quantity and will fit a straight line through them, so a genuinely ordered variable such as small, medium, large works and an unordered one such as city does not. Here is what that costs, measured.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import GradientBoostingRegressor

rng = np.random.default_rng(0)
n = 900
city = rng.integers(0, 3, n)                     # 0 Leeds, 1 Bristol, 2 Cardiff
effect = np.array([10.0, 0.0, 5.0])              # no monotonic order at all
y = 100 + effect[city] + rng.normal(0, 1, n)     # noise floor is 1.0

def rmse(a, b):
    return round(float(np.sqrt(np.mean((a - b) ** 2))), 3)

X_ord = city.reshape(-1, 1).astype(float)
X_oh = np.eye(3)[city]

print("linear, ordinal ", rmse(y, LinearRegression().fit(X_ord, y).predict(X_ord)))
print("linear, one-hot ", rmse(y, LinearRegression().fit(X_oh, y).predict(X_oh)))
print("tree,   ordinal ", rmse(y, GradientBoostingRegressor(random_state=0)
                               .fit(X_ord, y).predict(X_ord)))

# linear, ordinal  3.643    it must draw a straight line through 10, 0, 5
# linear, one-hot  0.974    exactly the noise floor
# tree,   ordinal  0.974    a tree can split 0 | 1 | 2 however it likes

Target encoding replaces each category with the average outcome for that category, which compresses ten thousand cities into one informative column. It is the strongest encoding on high-cardinality data and it invites two distinct failures, so it needs two separate defences.

The first is rare categories. A city with two rows, both of which happened to convert, gets an encoded value of 1.0, and the model learns that this city always converts on the evidence of two rows. Shrink towards the overall rate, with the shrinkage weakening as the count grows.

def smooth_target_encode(sum_y, count, prior, k=20):
    """k is how many rows of evidence it takes before the category's own rate wins."""
    return (sum_y + prior * k) / (count + k)

prior = 0.10                                    # the overall conversion rate
for count, rate in [(2, 1.0), (20, 0.5), (500, 0.5)]:
    print(count, rate, round(smooth_target_encode(rate * count, count, prior), 4))

# 2   1.0  0.1818    two conversions from two rows: barely moves off the prior
# 20  0.5  0.3       halfway, because count equals k
# 500 0.5  0.4846    five hundred rows: the category's own rate, essentially

The second failure is leakage, and it is worse because it is invisible. Each row's own label is inside the average used to encode it, so the feature contains the answer. On a category with three rows, a third of the encoding is the target. The fix is to compute each row's encoding from other folds only, which scikit-learn's TargetEncoder does internally during fit_transform. The size of the problem is easiest to see on data with no signal whatsoever.

import numpy as np

rng = np.random.default_rng(3)
n = 600
cat = rng.integers(0, 200, n)         # 200 categories, about 3 rows each
y = rng.normal(0, 1, n)               # pure noise: nothing can predict it
folds = np.arange(n) % 5

naive = np.array([y[cat == c].mean() for c in cat])       # includes each row itself

oof = np.empty(n)
for f in range(5):
    tr = folds != f
    means = {}
    for c in range(200):
        m = tr & (cat == c)
        means[c] = y[m].mean() if m.any() else y[tr].mean()
    oof[folds == f] = [means[c] for c in cat[folds == f]]

print(round(float(np.corrcoef(naive, y)[0, 1]), 3))   # 0.536
print(round(float(np.corrcoef(oof, y)[0, 1]), 3))     # -0.023

Two cheaper encodings are worth knowing because they carry no leakage risk at all. Frequency encoding replaces a category with how often it appears, which is one column, needs no labels, and is surprisingly useful whenever rarity itself is informative — an unusual browser, an uncommon merchant. Hashing maps the category string into a fixed number of columns through a hash function, which bounds the width regardless of how many categories arrive, at the cost of collisions that you cannot inspect afterwards.

Scaling#

Linear models, neural networks and anything distance-based need features on comparable scales, or a feature measured in thousands dominates one measured in units. Tree models do not care, because they split on thresholds and are invariant to monotonic transformations.

Dominates is not a figure of speech. Two customers, twenty years apart in age and a hundred pounds apart in income, are almost the same person to a distance calculation until the features are put on the same footing.

import numpy as np

a = np.array([25.0, 50_000])          # age, income
b = np.array([45.0, 50_100])

print(round(float(np.linalg.norm(a - b)), 2))          # 101.98

sigma = np.array([10.0, 15_000])                       # the spread of each feature
print(((a - b) / sigma).round(4))                      # [-2.     -0.0067]
print(round(float(np.linalg.norm((a - b) / sigma)), 2))  # 2.0

Model                        Scaling        Why
linear and logistic          matters        with regularisation, it decides
                                            which coefficients get penalised
k-nearest neighbours, SVM    matters        the distance is the model
k-means, PCA                 matters        variance is what they maximise
neural networks              matters        inputs on wild scales make the
                                            gradients wild too
decision trees, forests,     no effect      splits compare against a threshold,
gradient boosting                           which any monotonic map preserves
naive Bayes                  no effect      per-feature distributions

• Standardise, subtracting the mean and dividing by the standard deviation, is the default. It assumes nothing about the range and is what most models expect.
• Min-max, mapping to zero and one, when a bounded range is required — an image pixel, an input to a layer that expects it. It is dominated by the extremes, so one outlier compresses everything else.
• Scaling by the median and the interquartile range, which scikit-learn calls RobustScaler, when outliers are real and you do not want them setting the scale.
• A log transform before any of these, for anything heavy-tailed such as income, prices or counts. It often does more for a linear model than the choice of scaler.
• Fit the scaler on the training fold and apply it to validation. Fitting on everything is the most common leak in the whole of this section.

Constructing features#

This is where most of the gain is on tabular problems: encoding something you know about the domain that the raw columns do not express.

df["order_hour"] = df.created_at.dt.hour
df["is_weekend"] = df.created_at.dt.dayofweek >= 5
df["days_since_last"] = (df.created_at - df.prev_order).dt.days
df["items_per_order"] = df.item_count / df.order_count
df["above_customer_average"] = df.total > df.customer_mean_total

The reason those five help is the same in each case: they state something the raw columns only imply. A model given created_at and prev_order can in principle learn that the gap matters, but it has to discover subtraction from the data, using up capacity and examples to arrive at a fact you already knew. Handing it the gap directly is the whole of feature engineering in one sentence.

Raw columns you have             What they only imply
two timestamps                   the gap between them, and whether it is growing
a total and a count              the average per item, which is comparable
                                 across customers where the total is not
a value and a group average      whether this one is unusual for its group
a postcode and a postcode        the distance, which is the thing that matters
a count this month, last month   the trend, which predicts better than either
a date                           weekday, holiday, payday, end of quarter
a price and a list price         the discount, in percent

Three habits make this go faster. Ask someone who does the job manually what they look at, and build that. Look at the rows the model gets most wrong and ask what a person would have noticed about them. And build features in families rather than one at a time — a count over seven, thirty and ninety days, and the ratios between those windows — because the right window is an empirical question and testing three costs almost nothing.

Leakage#

The test is whether the feature would genuinely be available, with that value, at the moment you need to predict. If you cannot be sure, compute it inside the training fold and see whether the score collapses.

Relevance to LLM work#

Feature engineering largely disappears when inputs are text and the model is fixed, which is why it sits on the ML Engineer track and not the AI Engineer one. The leakage discipline transfers directly, though: an eval set contaminated with information the system would not have in production fails in exactly the same way.

On tabular problems the features usually decide the result more than the model does. This section builds the ones that keep earning their place: encodings for categories with thousands of values, the several signals hiding inside a timestamp, behaviour summaries over a window, and missingness treated as information rather than a gap to fill. One discipline runs through all of it, which is that every feature must be computed from data that existed at the moment of prediction. Break it and validation looks superb while production does not.

Categories with thousands of values#

One-hot encoding a column with ten thousand cities makes ten thousand mostly empty columns. Target encoding replaces each category with the average outcome for that category. It is powerful and leaks badly if done naively, because each row's own label is inside its encoding. Compute it out of fold: each row's value comes from other folds only.

from sklearn.preprocessing import TargetEncoder   # scikit-learn 1.3+

enc = TargetEncoder()                 # cross-fits internally during fit_transform
X_train["city_enc"] = enc.fit_transform(X_train[["city"]], y_train)
X_test["city_enc"]  = enc.transform(X_test[["city"]])

Dates carry several signals#

df["hour"]         = df.ts.dt.hour
df["weekday"]      = df.ts.dt.weekday
df["is_month_end"] = df.ts.dt.is_month_end
df["days_since_signup"] = (df.ts - df.signup_ts).dt.days

Aggregates, as of the moment of prediction#

The strongest features are often behaviour summaries: orders in the last 30 days, average basket size, days since last login. The trap is computing them with data from after the moment you are predicting. A churn model trained with 'logins in the last 30 days' computed today, for customers who churned months ago, has seen the future.

# For each prediction row (customer, as_of), count orders in the 30 days before as_of
m = orders.merge(rows, on="customer_id")
in_window = (m.order_ts < m.as_of) & (m.order_ts >= m.as_of - pd.Timedelta(days=30))
feats = m[in_window].groupby(["customer_id", "as_of"]).size().rename("orders_30d")

The difference between a point-in-time feature and the version people write first is easiest to see with everything stripped away. Both count the same customer's orders; only one of them changes depending on when you ask.

orders = [("c1", 1), ("c1", 5), ("c1", 40), ("c1", 71), ("c2", 3)]

def orders_in_window(orders, customer, as_of, days=30):
    return sum(1 for c, t in orders if c == customer and as_of - days <= t < as_of)

def orders_all_time(orders, customer):
    return sum(1 for c, _ in orders if c == customer)

for as_of in (10, 45, 75):
    print(as_of, orders_in_window(orders, "c1", as_of), orders_all_time(orders, "c1"))

# 10 2 4
# 45 1 4
# 75 1 4

Available at the moment of prediction is stricter than it sounds, in two ways people miss. A value that existed but had not yet arrived in your warehouse is not available: if payments settle overnight, the feature for a prediction made at nine in the morning cannot use last night's payments. And a value that was later corrected is not the value you had: if the warehouse overwrites a row when a refund is processed, reading it today gives you the corrected figure, not the one the model would have seen. Tables that keep a history of what was known when — slowly changing dimensions — are the usual answer, and their absence is the usual reason a backtest cannot be trusted.

Build one function that produces the features for a given entity and a given moment, and call it from both training and serving. Two implementations of the same feature will diverge, quietly, within a quarter, and the symptom is a model that scores well offline and does nothing in production with nothing in the logs to explain it.

Missing is a value#

A missing income field may mean the customer declined to answer, which can predict the outcome. Add an indicator column for missingness before filling the gap. Tree models such as gradient boosting can often handle missing values natively.

How much that matters depends on why the value is missing, and there are three cases worth telling apart. Missing at random, such as a sensor that drops readings independently of what it is reading: impute and move on. Missing for a reason visible in other columns, such as a field only asked of business accounts: impute within group, and the indicator is redundant with a column you already have. And missing because of the thing you are predicting, such as an income field that people with low incomes decline to fill in: the imputed value is worse than useless and the indicator is the feature.

import numpy as np

def roc_auc(y, s):
    order = np.argsort(s)
    ranks = np.empty(len(s))
    ranks[order] = np.arange(1, len(s) + 1)
    pos, neg = y.sum(), (1 - y).sum()
    return float((ranks[y == 1].sum() - pos * (pos + 1) / 2) / (pos * neg))

rng = np.random.default_rng(0)
n = 4000
risky = rng.random(n) < 0.3                                 # never observed
income = np.where(risky, rng.normal(20, 5, n), rng.normal(45, 10, n))
declined = risky & (rng.random(n) < 0.8)                    # risky people skip the field
y = (rng.random(n) < np.where(risky, 0.40, 0.08)).astype(int)

observed = np.where(declined, np.nan, income)
filled = np.where(np.isnan(observed), np.nanmedian(observed), observed)
is_missing = np.isnan(observed).astype(float)

print(round(roc_auc(y, -filled), 3))                        # 0.546
print(round(roc_auc(y, is_missing), 3))                     # 0.684
print(round(roc_auc(y, is_missing - filled / 100), 3))      # 0.711

Text as features#

For a text column, TF-IDF gives a strong, cheap baseline. Sentence embeddings from a pretrained model capture meaning and can be fed to any classifier as numeric features. Try TF-IDF first; it is often closer than people expect.

Before either, try the cheap structural features, because on short fields they frequently beat both. The length of the text, whether it contains a digit, a URL or a phone number, the count of capital letters, whether it is empty at all. A product description of four words and a product description of four hundred are different products, and no embedding of the words says that as directly as the length does.

The cost of each is worth knowing before you choose. TF-IDF produces tens of thousands of sparse columns, fits in seconds on a laptop, and can be inspected term by term when someone asks why. Embeddings produce a few hundred dense columns, need a model at serving time and therefore a latency budget and a version to pin, and cannot be explained to anyone. Take the second only when the first has been tried and the gap is worth the operational weight.

Which features help?#

Permutation importance shuffles one feature on the validation set and measures how much the score drops. It is more trustworthy than a tree's built-in importance, which favours features with many distinct values. Correlated features share credit, so a low score does not always mean useless.

That last caveat is the one that causes real damage. Two features carrying the same information each score near zero, because shuffling one leaves the other to answer the question, and a tidy-minded engineer removes both and loses the signal entirely. Where features are correlated, drop them one at a time and refit, or group them and permute the group together.

Importance also answers a different question from the one usually asked of it. It says what this model relies on, not what is predictive: a feature the model never learned to use scores zero, and so does a feature that genuinely does not matter. And a feature at the top with no plausible mechanism is a leak far more often than it is an insight, so the first thing to do with the top of the list is not to celebrate it but to ask when each of those values becomes known.

from sklearn.inspection import permutation_importance

r = permutation_importance(model, X_val, y_val, n_repeats=10, scoring="roc_auc")
for i in r.importances_mean.argsort()[::-1][:10]:
    print(f"{X_val.columns[i]:24} {r.importances_mean[i]:.4f}")