Retrieve broadly, then rank precisely

One retriever has to do two jobs that pull against each other: find everything relevant, and put the best of it first. A method fast enough to scan a million documents cannot afford to read the query and a document together, so it compares two independently-made vectors and is approximate by construction. The fix is to split the jobs. Stage one fetches fifty candidates and is judged only on whether the right answer is somewhere in them; stage two reads each candidate against the query properly and reorders. This section builds both halves, the fusion that joins two stage-one retrievers, and the arithmetic that decides how many candidates you can afford.

Why one stage is not enough#

The two stages are measured by different numbers, and this is the part that is worth getting straight before any code. Stage one is judged on recall at k: of the documents that should have been found, how many are in the fifty I fetched? Getting them in a good order is not its job. Stage two is judged on precision at the top: given those fifty, are the best three now first? Optimising stage one for ordering is wasted effort, and measuring stage two on recall cannot show anything, since it can only reorder what it was given. Tuning the wrong number for the wrong stage is the most common way this architecture is got wrong.

Hybrid search: the lexical half#

Embeddings are weak at exactly the things users type most literally: error codes, product names, identifiers, rare acronyms. A vector for "429" is not distinctive, because the model never had a reason to make it so. Keyword search is excellent at precisely those and hopeless at paraphrase, so production retrieval usually runs both. The keyword half is BM25, which is short enough to write out, and worth writing out once because the two constants in it are the whole idea.

import math, re
from collections import Counter

def tokenize(text):
    return re.findall(r"[a-z0-9]+", text.lower())

class BM25:
    def __init__(self, docs, k1=1.5, b=0.75):
        self.k1, self.b = k1, b                  # k1: saturation.  b: length penalty
        self.docs = [tokenize(d) for d in docs]
        self.lengths = [len(d) for d in self.docs]
        self.avg_len = sum(self.lengths) / len(self.docs)
        self.freqs = [Counter(d) for d in self.docs]
        df = Counter(t for d in self.docs for t in set(d))
        n = len(self.docs)
        # Rare terms count for more. The halves keep it finite for a term in every doc.
        self.idf = {t: math.log((n - c + 0.5) / (c + 0.5) + 1) for t, c in df.items()}

    def score(self, query, i):
        total = 0.0
        for t in tokenize(query):
            if t not in self.freqs[i]:
                continue
            f = self.freqs[i][t]
            norm = 1 - self.b + self.b * self.lengths[i] / self.avg_len
            total += self.idf[t] * f * (self.k1 + 1) / (f + self.k1 * norm)
        return total

    def top(self, query, k=3):
        order = sorted(range(len(self.docs)), key=lambda i: -self.score(query, i))
        return [(i, round(self.score(query, i), 3)) for i in order[:k]]

docs = [
    "The API rate limit is 100 requests per minute per API key",
    "Throttling behaviour and what a 429 response means for clients",
    "Reset your password from the account settings page",
    "Billing runs monthly and invoices are issued on the first",
    "A 429 is returned when you exceed the rate limit",
]
bm = BM25(docs)
print(bm.top("429 rate limit"))
print("idf 429:", round(bm.idf["429"], 3), " idf the:", round(bm.idf["the"], 3))

# [(4, 2.626), (0, 1.606), (1, 0.875)]
# idf 429: 0.875  idf the: 0.288

The two constants are worth understanding rather than copying. k1 controls saturation: a document mentioning your term ten times is more relevant than one mentioning it twice, but not five times more, so the score flattens. b controls length: without it, long documents win by accident because they contain more of everything. Defaults of 1.5 and 0.75 are used almost universally and are rarely the thing worth tuning.

Fusing two rankings#

Now you have two ranked lists whose scores mean different things — BM25 is unbounded and starts at zero, cosine sits between minus one and one. Adding them lets whichever has the larger range decide everything, and normalising them per query is fragile because the ranges move with the query. Reciprocal rank fusion sidesteps this by throwing the scores away and using only the positions.

def rrf(rankings, k=60):
    """Each list votes by position. Scores are never compared."""
    scores = {}
    for ranking in rankings:
        for pos, doc in enumerate(ranking):
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + pos + 1)
    return sorted(scores, key=scores.get, reverse=True)

lexical  = ["d4", "d1", "d7", "d2"]      # BM25 order
semantic = ["d7", "d4", "d9", "d1"]      # embedding order
print(rrf([lexical, semantic]))

# ['d4', 'd7', 'd1', 'd9', 'd2']
#   d4: lex 1st (0.01639) + sem 2nd (0.01613)   <- in both, so it rises
#   d7: lex 3rd (0.01587) + sem 1st (0.01639)
#   d9: sem 3rd only (0.01587)                  <- in one, so it sinks

Cross-encoder reranking#

Stage two can afford what stage one cannot: putting the query and the document into the model together, so that attention runs across both and the score reflects them jointly rather than comparing two vectors made in ignorance of each other. That is a cross-encoder, and it is more accurate for the same reason it is slower — nothing can be precomputed, because the pair does not exist until the query arrives. A bi-encoder embeds a million documents once; a cross-encoder must do one forward pass per candidate, per query.

bi-encoder  (stage one)          cross-encoder (stage two)
  embed(query)  -> vector           model(query, document) -> score
  embed(doc)    -> vector           both texts, one pass, attention across
  cosine between them               nothing precomputable
  documents embedded once           one pass per candidate per query
  millions of docs, milliseconds    tens of candidates, tens of ms each

Which makes the candidate count a latency decision rather than a quality one. At a rough 8 ms per pair on a small cross-encoder, reranking 10 candidates adds 80 ms, 50 adds 400 ms and 100 adds 800 ms — to every query, on top of retrieval and generation. Measure the per-pair cost of your own model and multiply, because that product is what sets your candidate count. Most systems land between 20 and 50, and the way to choose is to measure recall at k on your own data and take the smallest k where it stops climbing.

Measuring the improvement#

The honest test is a set of queries with known correct documents — a hundred is enough to see a real effect — scored before and after the reranker, with the stages measured separately.

stage one, recall@50      is the answer in the candidates at all?
stage two, MRR            1/rank of the first correct document, averaged
stage two, precision@3    of the three you will actually show, how many are right
end to end                answer quality, which is the only one that pays

And measure the latency in the same run. A reranker is a permanent tax on every query, so the comparison that decides whether it ships is the quality gain against the added milliseconds and the added cost — not the quality gain alone. If the gain is small and the budget is tight, better chunking and a query rewrite are usually the cheaper purchase, and both are covered in the sections around this one.