Matching people to items

Recommenders are among the largest machine learning systems in production anywhere, and the central constraint is one you have already met in retrieval: you cannot score a million items for every request. This section covers the two-stage shape that follows — cheap candidate generation, then expensive ranking — the signals you can match on, and what to do about a user or an item with no history at all. It also covers the uncomfortable part, that the system trains on what it previously recommended, so offline metrics can rise while the experience narrows.

What you have about this user or item      What can actually score it
years of their clicks                      collaborative: people like them
three clicks, from today                   their session, plus popularity
nothing at all, new account                popularity, by locale and device
item live for a year                       collaborative: people who took it
item uploaded ten minutes ago              content: its own features
item nobody has ever been shown            exploration, or it stays invisible

Two stages#

You cannot score a million items per request. Candidate generation narrows to a few hundred cheaply; ranking scores those precisely. The same two-stage shape as retrieval in a RAG system, for the same reason.

millions of items
      |  candidate generation, cheap and broad
      v
  ~500 candidates
      |  ranking, expensive and precise
      v
  20 shown

The reason is arithmetic rather than taste. Suppose the ranking model takes 200 microseconds per item, which is fast for anything with real features in it. Scoring two million items is 400 seconds for one request. Scoring five hundred is a tenth of a second, and that fits in a page load. There is no model that is both accurate enough to rank with and cheap enough to run over the whole catalogue, so you need two models with different jobs.

stage         items in   items out   budget   what it may use
retrieval     2,000,000  ~500        10 ms    one vector per item, ANN index
ranking       500        ~50         50 ms    full features, gradient boosting
re-ranking    50         20          5 ms     diversity, freshness, rules

The two stages fail differently and that difference decides what you work on. Retrieval fails by omission: a good item never reaches the ranker and no amount of ranking improvement can recover it, so retrieval is measured by recall — did the set of five hundred contain the things this user would have liked. Ranking fails by ordering: the right items were there and the wrong one went first, so ranking is measured by position-aware metrics. If your feed is dull, find out which of the two it is before touching either, because tuning the ranker on candidates that never contained anything interesting is a month spent reordering the same twenty items.

Collaborative filtering#

The collaborative signal is that people who behaved alike in the past will behave alike next. It needs nothing about what the items are — no titles, no genres, no descriptions — which is its defining strength: it discovers that two items belong together because the same people take both, not because somebody tagged them the same.

The item-to-item form is the one worth being able to write out. Build a matrix of users against items, treat each item as its column, and measure how similar two columns are. Recommending then means summing the similarities of everything the user already took.

import numpy as np

# Rows are users, columns are items. A one means that user took that item.
R = np.array([
    [1, 1, 0, 0, 0],
    [1, 1, 1, 0, 0],
    [0, 1, 1, 0, 0],
    [0, 0, 0, 1, 1],
    [0, 0, 0, 1, 1],
], dtype=float)

norms = np.linalg.norm(R, axis=0)
S = (R.T @ R) / np.outer(norms, norms)      # cosine similarity between items
np.fill_diagonal(S, 0)                      # an item is not its own recommendation

def recommend(user, k=1):
    scores = S @ R[user]                    # sum similarity to what they took
    scores[R[user] > 0] = -np.inf           # never recommend what they have
    return np.argsort(-scores)[:k], scores

picks, scores = recommend(0)
print(np.round(scores, 2))   # [-inf -inf 1.32 0.   0.  ]
print(picks)                 # [2]  -- item 2, which users 2 and 3 took alongside item 1

Three things it fails at, all of them structural rather than fixable by tuning. It cannot score an item nobody has taken, which is every item on its first day. It is biased towards the popular, because a column with ten thousand ones overlaps with everything and a column with four ones overlaps with almost nothing, so the tail stays invisible even when it is exactly right for someone. And it has no idea why, so when the product team asks for an explanation next to each recommendation there is nothing honest to write.

Content-based filtering#

The content signal is that this item resembles things the user already liked, where resemblance is computed from the item itself: its text, its images, its category, its creator, its price. The user is represented as a summary of the items they took, and scoring a new item means comparing it to that summary. An embedding model does this well for anything with text or images attached.

What it buys you is the entire cold-item problem solved. An item uploaded ten minutes ago has features, so it has an embedding, so it can be retrieved. That is not a small thing: for a news site, a marketplace or a video platform, most of what matters was created this week.

What it fails at is everything collaborative filtering is for. It recommends more of the same, because more of the same is what similarity means, so a user who read three articles about interest rates gets a fourth and never the one thing outside that they would actually have loved. It is capped by the quality of the features, and item metadata in the real world is entered by whoever uploaded it. And it cannot find the non-obvious pairing — the two products that have nothing in common except that the same people buy both — which is exactly the recommendation that feels like magic and is worth the most.

                       Collaborative              Content
needs                  interaction history        item features
new item               cannot score it            fine, day one
new user               cannot score them          fine, after one action
long tail              buried by popularity       reachable
surprise               finds non-obvious pairs    more of the same
explainable            not really                 yes, by feature
degrades when          the catalogue turns over   metadata is poor

Cold start#

A new user has no history and a new item has no interactions, so collaborative filtering has nothing to work with. There are four cases and they need different answers, which is why one paragraph about cold start is never enough.

• New user. Fall back to popularity conditioned on whatever you do know: country, language, device, referrer. Then switch to their session as soon as it exists, because two clicks in the current session is a stronger signal than a demographic prior. Asking them to pick three interests at sign-up works, and costs you some of them at the door.
• New item. Retrieve it by its content embedding, and give it a deliberate budget of impressions so it acquires the interaction history it needs. Without the budget it is invisible forever: no impressions means no clicks, which means no collaborative signal, which means no impressions.
• New system, no data at all. Ship content-based retrieval plus popularity and start logging. Matrix factorisation on three weeks of interactions is worse than sorting by popularity, and the logs are the asset you are actually building.
• The returning user who has been away a year. Technically warm, practically cold: their taste has moved and the last thing they clicked is about a job they no longer have. Weight recent interactions much more heavily than old ones, and treat a long gap as a partial reset.

The mechanism that makes the new-item case work is shrinkage. An item with two clicks from five impressions does not have a forty percent click rate; it has almost no information, and a greedy ranker that believes the raw ratio will put it at the top of everyone's feed on the strength of two clicks. Pull the estimate towards the catalogue average, with the pull weakening as evidence accumulates.

def smoothed_rate(clicks, impressions, prior_rate, strength=200):
    """Shrink towards the catalogue average. strength is how many impressions
    of evidence it takes before the item's own data dominates the prior."""
    return (clicks + prior_rate * strength) / (impressions + strength)

prior = 0.04                                  # the catalogue-wide click rate
for imps, clicks in [(5, 2), (200, 8), (5000, 150), (5000, 400)]:
    raw = clicks / imps
    print(imps, clicks, round(raw, 3), round(smoothed_rate(clicks, imps, prior), 4))

# 5    2   raw 0.4    smoothed 0.0488   two clicks prove nothing
# 200  8   raw 0.04   smoothed 0.04     average, and known to be average
# 5000 150 raw 0.03   smoothed 0.0304   genuinely below average
# 5000 400 raw 0.08   smoothed 0.0785   genuinely good, and allowed to say so

Feedback loops#

It is worth watching that happen rather than taking it on trust. Below, two hundred items have a fixed true click rate that the system never observes. One run always shows the current best estimates; the other gives two of its ten slots to items chosen at random. Everything else is identical.

import numpy as np

def simulate(explore_slots, rounds=400, n_items=200, slots=10, seed=0):
    rng = np.random.default_rng(seed)
    quality = rng.beta(2, 20, n_items)          # true click rates, never observed
    clicks, shown = np.zeros(n_items), np.zeros(n_items)
    for _ in range(rounds):
        estimate = (clicks + 1) / (shown + 25)  # smoothed, as above
        picks = list(np.argsort(-estimate)[: slots - explore_slots])
        if explore_slots:
            picks += list(rng.choice(n_items, explore_slots, replace=False))
        picks = np.array(picks)
        shown[picks] += 1
        clicks[picks] += rng.random(len(picks)) < quality[picks]
    return quality, shown, clicks

for explore in (0, 2):
    quality, shown, clicks = simulate(explore)
    top = np.argsort(-shown)[:10]
    print(
        "explore", explore,
        "| distinct items shown", int((shown > 0).sum()),
        "| mean true rate of the 10 most-shown", round(quality[top].mean(), 4),
        "| clicks", int(clicks.sum()),
    )

# explore 0 | distinct 182 | mean true rate of the 10 most-shown 0.1086 | clicks 481
# explore 2 | distinct 195 | mean true rate of the 10 most-shown 0.1562 | clicks 643
# The best ten items in the catalogue average 0.2302, so neither run found them.

Offline metrics mislead#

Offline you can only measure whether you would have predicted what the user actually did, under the old system. A genuinely better recommender shows different things, which offline evaluation scores as wrong. This is why recommender work depends on live experiments more than most machine learning.

# The old system only ever showed this user items 0 to 49.
exposed = set(range(50))
would_like = {3, 7, 41, 120, 155}          # unobservable: what they actually enjoy
logged = would_like & exposed              # {3, 7, 41}: all the offline test can see

def offline_recall(recs, k=5):
    return len(set(recs[:k]) & logged) / len(logged)

old_model = [3, 7, 41, 12, 19]
new_model = [3, 120, 155, 7, 88]           # surfaces two items they would love
print(round(offline_recall(old_model), 2), round(offline_recall(new_model), 2))
# 1.0 0.67

That does not make offline evaluation useless, and treating it as useless is the opposite mistake. It is reliable for the things that do not depend on what was shown: whether a change broke something, whether retrieval recall against known-relevant items went up, whether latency regressed, whether a feature has predictive value at all. Use it as a filter that kills bad ideas cheaply. It is unreliable for the size of an improvement, for anything involving items the old system rarely showed, and for the ordering of two models that are close — which is precisely when someone wants an answer from it.

• Offline predicts, roughly: this change is broken, this change does nothing, retrieval now covers items it used to miss.
• Offline does not predict: how much engagement moves, whether the feed feels repetitive, whether the new items get discovered, what happens in week three when the model has retrained on its own output.
• The last one is the one that surprises people. Every metric can improve in a two-week test and the system still degrade over a quarter, because the training data is now the product of the thing being measured.