Features for real tabular problems

On tabular problems the features usually decide the result more than the model does. This section builds the ones that keep earning their place: encodings for categories with thousands of values, the several signals hiding inside a timestamp, behaviour summaries over a window, and missingness treated as information rather than a gap to fill. One discipline runs through all of it, which is that every feature must be computed from data that existed at the moment of prediction. Break it and validation looks superb while production does not.

Categories with thousands of values#

One-hot encoding a column with ten thousand cities makes ten thousand mostly empty columns. Target encoding replaces each category with the average outcome for that category. It is powerful and leaks badly if done naively, because each row's own label is inside its encoding. Compute it out of fold: each row's value comes from other folds only.

from sklearn.preprocessing import TargetEncoder   # scikit-learn 1.3+

enc = TargetEncoder()                 # cross-fits internally during fit_transform
X_train["city_enc"] = enc.fit_transform(X_train[["city"]], y_train)
X_test["city_enc"]  = enc.transform(X_test[["city"]])

Dates carry several signals#

df["hour"]         = df.ts.dt.hour
df["weekday"]      = df.ts.dt.weekday
df["is_month_end"] = df.ts.dt.is_month_end
df["days_since_signup"] = (df.ts - df.signup_ts).dt.days

Aggregates, as of the moment of prediction#

The strongest features are often behaviour summaries: orders in the last 30 days, average basket size, days since last login. The trap is computing them with data from after the moment you are predicting. A churn model trained with 'logins in the last 30 days' computed today, for customers who churned months ago, has seen the future.

# For each prediction row (customer, as_of), count orders in the 30 days before as_of
m = orders.merge(rows, on="customer_id")
in_window = (m.order_ts < m.as_of) & (m.order_ts >= m.as_of - pd.Timedelta(days=30))
feats = m[in_window].groupby(["customer_id", "as_of"]).size().rename("orders_30d")

The difference between a point-in-time feature and the version people write first is easiest to see with everything stripped away. Both count the same customer's orders; only one of them changes depending on when you ask.

orders = [("c1", 1), ("c1", 5), ("c1", 40), ("c1", 71), ("c2", 3)]

def orders_in_window(orders, customer, as_of, days=30):
    return sum(1 for c, t in orders if c == customer and as_of - days <= t < as_of)

def orders_all_time(orders, customer):
    return sum(1 for c, _ in orders if c == customer)

for as_of in (10, 45, 75):
    print(as_of, orders_in_window(orders, "c1", as_of), orders_all_time(orders, "c1"))

# 10 2 4
# 45 1 4
# 75 1 4

Available at the moment of prediction is stricter than it sounds, in two ways people miss. A value that existed but had not yet arrived in your warehouse is not available: if payments settle overnight, the feature for a prediction made at nine in the morning cannot use last night's payments. And a value that was later corrected is not the value you had: if the warehouse overwrites a row when a refund is processed, reading it today gives you the corrected figure, not the one the model would have seen. Tables that keep a history of what was known when — slowly changing dimensions — are the usual answer, and their absence is the usual reason a backtest cannot be trusted.

Build one function that produces the features for a given entity and a given moment, and call it from both training and serving. Two implementations of the same feature will diverge, quietly, within a quarter, and the symptom is a model that scores well offline and does nothing in production with nothing in the logs to explain it.

Missing is a value#

A missing income field may mean the customer declined to answer, which can predict the outcome. Add an indicator column for missingness before filling the gap. Tree models such as gradient boosting can often handle missing values natively.

How much that matters depends on why the value is missing, and there are three cases worth telling apart. Missing at random, such as a sensor that drops readings independently of what it is reading: impute and move on. Missing for a reason visible in other columns, such as a field only asked of business accounts: impute within group, and the indicator is redundant with a column you already have. And missing because of the thing you are predicting, such as an income field that people with low incomes decline to fill in: the imputed value is worse than useless and the indicator is the feature.

import numpy as np

def roc_auc(y, s):
    order = np.argsort(s)
    ranks = np.empty(len(s))
    ranks[order] = np.arange(1, len(s) + 1)
    pos, neg = y.sum(), (1 - y).sum()
    return float((ranks[y == 1].sum() - pos * (pos + 1) / 2) / (pos * neg))

rng = np.random.default_rng(0)
n = 4000
risky = rng.random(n) < 0.3                                 # never observed
income = np.where(risky, rng.normal(20, 5, n), rng.normal(45, 10, n))
declined = risky & (rng.random(n) < 0.8)                    # risky people skip the field
y = (rng.random(n) < np.where(risky, 0.40, 0.08)).astype(int)

observed = np.where(declined, np.nan, income)
filled = np.where(np.isnan(observed), np.nanmedian(observed), observed)
is_missing = np.isnan(observed).astype(float)

print(round(roc_auc(y, -filled), 3))                        # 0.546
print(round(roc_auc(y, is_missing), 3))                     # 0.684
print(round(roc_auc(y, is_missing - filled / 100), 3))      # 0.711

Text as features#

For a text column, TF-IDF gives a strong, cheap baseline. Sentence embeddings from a pretrained model capture meaning and can be fed to any classifier as numeric features. Try TF-IDF first; it is often closer than people expect.

Before either, try the cheap structural features, because on short fields they frequently beat both. The length of the text, whether it contains a digit, a URL or a phone number, the count of capital letters, whether it is empty at all. A product description of four words and a product description of four hundred are different products, and no embedding of the words says that as directly as the length does.

The cost of each is worth knowing before you choose. TF-IDF produces tens of thousands of sparse columns, fits in seconds on a laptop, and can be inspected term by term when someone asks why. Embeddings produce a few hundred dense columns, need a model at serving time and therefore a latency budget and a version to pin, and cannot be explained to anyone. Take the second only when the first has been tried and the gap is worth the operational weight.

Which features help?#

Permutation importance shuffles one feature on the validation set and measures how much the score drops. It is more trustworthy than a tree's built-in importance, which favours features with many distinct values. Correlated features share credit, so a low score does not always mean useless.

That last caveat is the one that causes real damage. Two features carrying the same information each score near zero, because shuffling one leaves the other to answer the question, and a tidy-minded engineer removes both and loses the signal entirely. Where features are correlated, drop them one at a time and refit, or group them and permute the group together.

Importance also answers a different question from the one usually asked of it. It says what this model relies on, not what is predictive: a feature the model never learned to use scores zero, and so does a feature that genuinely does not matter. And a feature at the top with no plausible mechanism is a leak far more often than it is an insight, so the first thing to do with the top of the list is not to celebrate it but to ask when each of those values becomes known.

from sklearn.inspection import permutation_importance

r = permutation_importance(model, X_val, y_val, n_repeats=10, scoring="roc_auc")
for i in r.importances_mean.argsort()[::-1][:10]:
    print(f"{X_val.columns[i]:24} {r.importances_mean[i]:.4f}")