Shaping inputs so the model can use them

A model cannot use a column that says "London" or a price that ranges from 3 to 300,000 alongside an age that ranges from 18 to 90. Feature engineering is the work of turning raw columns into numbers a model can actually learn from: encoding categories, putting different scales onto comparable footing, and building new columns that express something the raw ones only imply. The danger throughout is leakage — accidentally handing the model a column that quietly contains the answer.

What the column is                            Encode it as
under ~15 values, no order, linear model      one-hot
under ~15 values, no order, tree model        integer codes, or native categorical
a real order: small, medium, large            ordinal, in the right order
hundreds to thousands of values               target encoding, out of fold
thousands, and you cannot afford a lookup     hashing into a fixed width
you only need "how common is this"            frequency encoding
free text                                     TF-IDF, or sentence embeddings

Categories#

• One-hot: a column per category. Safe, and explodes with high cardinality.
• Ordinal: an integer per category. Only when the order is real, or you invent a ranking the model will believe.
• Target encoding: replace the category with the mean target for it. Powerful, and leaks badly unless computed inside the fold.
• Trees handle high-cardinality categories better than linear models do, which affects the choice.

One-hot is the default because it assumes nothing. Each category gets a column that is one when present and zero otherwise, so the model can learn an entirely separate effect for each with no ordering implied. What it costs is width. A column with ten thousand cities becomes ten thousand columns, almost all zero, which is fine for a linear model on a sparse matrix and genuinely bad for a tree: a tree has to find the one useful split among ten thousand near-empty columns, and each split isolates a single category rather than a group.

It also has a serving problem that only shows up in production. A category appears in live traffic that was not in the training data, and the encoder either raises or silently produces an all-zero row. Decide which you want, in writing: either map unknowns to a dedicated other column that existed at training time, or fail the request. Both are defensible; discovering the choice by reading a stack trace is not.

import numpy as np

def one_hot(values, categories):
    """Unknown values become an extra 'other' column, which must exist in training."""
    index = {c: i for i, c in enumerate(categories)}
    out = np.zeros((len(values), len(categories) + 1))
    for row, v in enumerate(values):
        out[row, index.get(v, len(categories))] = 1.0
    return out

cats = ["leeds", "bristol", "cardiff"]
print(one_hot(["bristol", "leeds", "york"], cats))
# [[0. 1. 0. 0.]
#  [1. 0. 0. 0.]
#  [0. 0. 0. 1.]]   <- york was never seen, and lands in 'other'

Ordinal encoding replaces each category with an integer, which is one column instead of ten thousand. The cost is that the integers mean something to some models. A linear model reads 0, 1, 2 as a quantity and will fit a straight line through them, so a genuinely ordered variable such as small, medium, large works and an unordered one such as city does not. Here is what that costs, measured.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import GradientBoostingRegressor

rng = np.random.default_rng(0)
n = 900
city = rng.integers(0, 3, n)                     # 0 Leeds, 1 Bristol, 2 Cardiff
effect = np.array([10.0, 0.0, 5.0])              # no monotonic order at all
y = 100 + effect[city] + rng.normal(0, 1, n)     # noise floor is 1.0

def rmse(a, b):
    return round(float(np.sqrt(np.mean((a - b) ** 2))), 3)

X_ord = city.reshape(-1, 1).astype(float)
X_oh = np.eye(3)[city]

print("linear, ordinal ", rmse(y, LinearRegression().fit(X_ord, y).predict(X_ord)))
print("linear, one-hot ", rmse(y, LinearRegression().fit(X_oh, y).predict(X_oh)))
print("tree,   ordinal ", rmse(y, GradientBoostingRegressor(random_state=0)
                               .fit(X_ord, y).predict(X_ord)))

# linear, ordinal  3.643    it must draw a straight line through 10, 0, 5
# linear, one-hot  0.974    exactly the noise floor
# tree,   ordinal  0.974    a tree can split 0 | 1 | 2 however it likes

Target encoding replaces each category with the average outcome for that category, which compresses ten thousand cities into one informative column. It is the strongest encoding on high-cardinality data and it invites two distinct failures, so it needs two separate defences.

The first is rare categories. A city with two rows, both of which happened to convert, gets an encoded value of 1.0, and the model learns that this city always converts on the evidence of two rows. Shrink towards the overall rate, with the shrinkage weakening as the count grows.

def smooth_target_encode(sum_y, count, prior, k=20):
    """k is how many rows of evidence it takes before the category's own rate wins."""
    return (sum_y + prior * k) / (count + k)

prior = 0.10                                    # the overall conversion rate
for count, rate in [(2, 1.0), (20, 0.5), (500, 0.5)]:
    print(count, rate, round(smooth_target_encode(rate * count, count, prior), 4))

# 2   1.0  0.1818    two conversions from two rows: barely moves off the prior
# 20  0.5  0.3       halfway, because count equals k
# 500 0.5  0.4846    five hundred rows: the category's own rate, essentially

The second failure is leakage, and it is worse because it is invisible. Each row's own label is inside the average used to encode it, so the feature contains the answer. On a category with three rows, a third of the encoding is the target. The fix is to compute each row's encoding from other folds only, which scikit-learn's TargetEncoder does internally during fit_transform. The size of the problem is easiest to see on data with no signal whatsoever.

import numpy as np

rng = np.random.default_rng(3)
n = 600
cat = rng.integers(0, 200, n)         # 200 categories, about 3 rows each
y = rng.normal(0, 1, n)               # pure noise: nothing can predict it
folds = np.arange(n) % 5

naive = np.array([y[cat == c].mean() for c in cat])       # includes each row itself

oof = np.empty(n)
for f in range(5):
    tr = folds != f
    means = {}
    for c in range(200):
        m = tr & (cat == c)
        means[c] = y[m].mean() if m.any() else y[tr].mean()
    oof[folds == f] = [means[c] for c in cat[folds == f]]

print(round(float(np.corrcoef(naive, y)[0, 1]), 3))   # 0.536
print(round(float(np.corrcoef(oof, y)[0, 1]), 3))     # -0.023

Two cheaper encodings are worth knowing because they carry no leakage risk at all. Frequency encoding replaces a category with how often it appears, which is one column, needs no labels, and is surprisingly useful whenever rarity itself is informative — an unusual browser, an uncommon merchant. Hashing maps the category string into a fixed number of columns through a hash function, which bounds the width regardless of how many categories arrive, at the cost of collisions that you cannot inspect afterwards.

Scaling#

Linear models, neural networks and anything distance-based need features on comparable scales, or a feature measured in thousands dominates one measured in units. Tree models do not care, because they split on thresholds and are invariant to monotonic transformations.

Dominates is not a figure of speech. Two customers, twenty years apart in age and a hundred pounds apart in income, are almost the same person to a distance calculation until the features are put on the same footing.

import numpy as np

a = np.array([25.0, 50_000])          # age, income
b = np.array([45.0, 50_100])

print(round(float(np.linalg.norm(a - b)), 2))          # 101.98

sigma = np.array([10.0, 15_000])                       # the spread of each feature
print(((a - b) / sigma).round(4))                      # [-2.     -0.0067]
print(round(float(np.linalg.norm((a - b) / sigma)), 2))  # 2.0

Model                        Scaling        Why
linear and logistic          matters        with regularisation, it decides
                                            which coefficients get penalised
k-nearest neighbours, SVM    matters        the distance is the model
k-means, PCA                 matters        variance is what they maximise
neural networks              matters        inputs on wild scales make the
                                            gradients wild too
decision trees, forests,     no effect      splits compare against a threshold,
gradient boosting                           which any monotonic map preserves
naive Bayes                  no effect      per-feature distributions

• Standardise, subtracting the mean and dividing by the standard deviation, is the default. It assumes nothing about the range and is what most models expect.
• Min-max, mapping to zero and one, when a bounded range is required — an image pixel, an input to a layer that expects it. It is dominated by the extremes, so one outlier compresses everything else.
• Scaling by the median and the interquartile range, which scikit-learn calls RobustScaler, when outliers are real and you do not want them setting the scale.
• A log transform before any of these, for anything heavy-tailed such as income, prices or counts. It often does more for a linear model than the choice of scaler.
• Fit the scaler on the training fold and apply it to validation. Fitting on everything is the most common leak in the whole of this section.

Constructing features#

This is where most of the gain is on tabular problems: encoding something you know about the domain that the raw columns do not express.

df["order_hour"] = df.created_at.dt.hour
df["is_weekend"] = df.created_at.dt.dayofweek >= 5
df["days_since_last"] = (df.created_at - df.prev_order).dt.days
df["items_per_order"] = df.item_count / df.order_count
df["above_customer_average"] = df.total > df.customer_mean_total

The reason those five help is the same in each case: they state something the raw columns only imply. A model given created_at and prev_order can in principle learn that the gap matters, but it has to discover subtraction from the data, using up capacity and examples to arrive at a fact you already knew. Handing it the gap directly is the whole of feature engineering in one sentence.

Raw columns you have             What they only imply
two timestamps                   the gap between them, and whether it is growing
a total and a count              the average per item, which is comparable
                                 across customers where the total is not
a value and a group average      whether this one is unusual for its group
a postcode and a postcode        the distance, which is the thing that matters
a count this month, last month   the trend, which predicts better than either
a date                           weekday, holiday, payday, end of quarter
a price and a list price         the discount, in percent

Three habits make this go faster. Ask someone who does the job manually what they look at, and build that. Look at the rows the model gets most wrong and ask what a person would have noticed about them. And build features in families rather than one at a time — a count over seven, thirty and ninety days, and the ratios between those windows — because the right window is an empirical question and testing three costs almost nothing.

Leakage#

The test is whether the feature would genuinely be available, with that value, at the moment you need to predict. If you cannot be sure, compute it inside the training fold and see whether the score collapses.

Relevance to LLM work#

Feature engineering largely disappears when inputs are text and the model is fixed, which is why it sits on the ML Engineer track and not the AI Engineer one. The leakage discipline transfers directly, though: an eval set contaminated with information the system would not have in production fails in exactly the same way.