Fixing the query, not just the ranking
By the end: You can improve retrieval from the query side, and measure each technique rather than stacking them.
- Turning a chat turn into a standalone question
- Several phrasings of one query
- Hypothetical answers as queries
- Breaking a question into parts
- Pulling filters out of the question
- Metrics: recall@k, MRR
New to reranking & hybrid search? Vector search is fast and rough: it finds passages about roughly the right subject, and often puts mediocre ones near the top. Reranking is a second pass over its output — take the fifty candidates it returned and score each one properly with a slower model that reads the question and the passage together, then keep the best five. Hybrid search is the companion idea: run an old-fashioned keyword search alongside the vector one and merge both lists, because keyword search finds exact strings like product codes and surnames that embeddings blur away. Why it matters
Retrieval failures often have nothing to do with the index. The user typed 'and what about for teams?', and no document resembles that. This section is about the step before the search: rewriting a conversational turn into a question that stands on its own, and the other query transformations that sometimes help, such as several phrasings, a hypothetical answer, or splitting a question in two. Each costs a model call and some latency, so the section also covers the measurements that tell you which ones to keep.
The query is often the problem#
In a conversation, the user's latest message is often 'and what about for teams?'. Embedding that finds nothing useful. The question only makes sense with the turns before it.
Rewrite into a standalone question#
History: "How do refunds work on the monthly plan?"
"Within 14 days, in full."
Latest: "and for teams?"
Rewritten: "How do refunds work on the monthly team plan?"One cheap model call before retrieval. For chat interfaces this is usually the single biggest improvement available.
Other techniques, each with a cost#
- Several phrasings: generate three versions of the query, search with each, fuse the results by rank. Helps when users phrase things unlike the documents.
- Hypothetical answer: ask a model to write a plausible answer, and search with that. Answers look more like documents than questions do. Risky when the model's guess points the search in the wrong direction.
- Decomposition: split 'compare the refund policy of plan A and plan B' into two searches.
- Filters: extract structured conditions such as a date range or product name, and apply them as metadata filters rather than hoping the embedding captures them.
A corpus small enough to argue with#
The rest of this section is measured rather than asserted, on fourteen chunks of a fictional pricing and policy document and five real-shaped follow-up turns. The retriever is a character n-gram TF-IDF index, which stands in for an embedding model so the run needs no network and gives the same numbers every time. It is kinder than a word-level index and less capable than a real embedding model, and the direction of every result below survives that substitution even though the exact figures would not.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
CHUNKS = {
"c1": ("Refunds on the monthly plan are issued in full within 14 days of purchase.", "monthly"),
"c2": ("Refunds on the annual plan are pro-rated after the first 30 days.", "annual"),
"c3": ("Team plans are billed per seat and refunds follow the monthly policy for each seat.", "team"),
"c4": ("Adding a seat to a team plan is charged pro-rata for the rest of the billing period.", "team"),
"c5": ("To reset your password, use the Forgot password link on the sign-in page.", "any"),
"c6": ("The free tier allows 60 requests per minute.", "free"),
"c7": ("The paid tier allows 6000 requests per minute and bursts to 12000.", "paid"),
"c8": ("Invoices are issued on the first of the month and sent to the billing contact.", "any"),
"c9": ("Single sign-on is available on team and enterprise plans.", "team"),
"c10": ("Cancelling a subscription stops renewal at the end of the current period.", "any"),
"c11": ("Data is deleted 30 days after an account is closed.", "any"),
"c12": ("Enterprise plans include a dedicated support channel and an uptime commitment.", "enterprise"),
"c13": ("The monthly plan costs 20 per user and is billed on the day you signed up.", "monthly"),
"c14": ("Support responses are within one business day on paid plans.", "paid"),
}
IDS = list(CHUNKS)
TEXTS = [CHUNKS[i][0] for i in IDS]
VEC = TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5)).fit(TEXTS)
M = VEC.transform(TEXTS)
def search(query, k=3, allowed=None):
sims = cosine_similarity(VEC.transform([query]), M)[0]
pool = [(i, s) for i, s in zip(IDS, sims) if allowed is None or i in allowed]
pool.sort(key=lambda p: -p[1])
return [i for i, _ in pool[:k]]
print("raw ", search("what about SSO?"))
print("rewritten ", search("Is single sign-on available on team plans?"))
# raw ['c4', 'c3', 'c10']
# rewritten ['c9', 'c13', 'c4']Measure every step#
Each technique adds latency and a model call. Some help your data and some do not. Build a set of real questions with the chunks that answer them, and measure after each change.
def recall_at_k(results, relevant, k=5):
return len(set(results[:k]) & relevant) / len(relevant)
def reciprocal_rank(results, relevant):
for i, r in enumerate(results, start=1):
if r in relevant:
return 1 / i
return 0.0 # average this over queries to get MRRTwo numbers, and they answer different questions. Recall at k asks whether the answering chunk reached the shortlist at all, which is the only thing that decides whether the model can possibly be right. MRR asks how far up it landed, which decides whether it survives a context budget that keeps three chunks. A change can move one and not the other, and knowing which it moved tells you whether to keep it.
QUERIES = [
{"raw": "and for teams?",
"rewritten": "How do refunds work on the team plan?",
"relevant": {"c3"}},
{"raw": "and on the paid one?",
"rewritten": "What is the rate limit on the paid tier?",
"relevant": {"c7"}},
{"raw": "what about SSO?",
"rewritten": "Is single sign-on available on team plans?",
"relevant": {"c9"}},
{"raw": "and if I add one mid-month?",
"rewritten": "What happens if I add a seat to a team plan mid-month?",
"relevant": {"c4"}},
{"raw": "how long do you keep it?",
"rewritten": "How long is data kept after an account is closed?",
"relevant": {"c11"}},
]
def evaluate(get_results, label, calls):
"""Every variant returns ten candidates, so the columns are comparable."""
rec = sum(recall_at_k(get_results(q), q["relevant"], k=3) for q in QUERIES) / len(QUERIES)
mrr = sum(reciprocal_rank(get_results(q), q["relevant"]) for q in QUERIES) / len(QUERIES)
print(f"{label:<28}{rec:>9.2f}{mrr:>7.2f}{calls:>8}")
return rec, mrr
print(f"{'query sent to the index':<28}{'recall@3':>9}{'MRR':>7}{'calls':>8}")
evaluate(lambda q: search(q["raw"], k=10), "the raw turn", 0)
evaluate(lambda q: search(q["rewritten"], k=10), "rewritten to standalone", 1)
# query sent to the index recall@3 MRR calls
# the raw turn 0.40 0.35 0
# rewritten to standalone 1.00 1.00 1Several phrasings, fused#
The idea is that users phrase things unlike the documents do, so generating two or three alternative phrasings and fusing the rankings covers more of the space. Fusing by rank rather than by score is what makes the merge meaningful, and it is the same reciprocal rank fusion the two-stage retrieval section uses to join a keyword and a vector search.
PHRASINGS = {
"How do refunds work on the team plan?": [
"team plan refund policy",
"can a team plan be refunded per seat",
],
"What is the rate limit on the paid tier?": [
"paid tier requests per minute",
"how many requests can a paid account make",
],
"Is single sign-on available on team plans?": [
"SSO support for teams",
"does the team plan have single sign-on",
],
"What happens if I add a seat to a team plan mid-month?": [
"adding a seat part way through a billing period",
"pro-rata charge for a new seat",
],
"How long is data kept after an account is closed?": [
"data retention after account closure",
"when is data deleted",
],
}
def rrf(rankings, k=60):
scores = {}
for ranking in rankings:
for pos, doc in enumerate(ranking):
scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + pos + 1)
return sorted(scores, key=scores.get, reverse=True)
def multi(q):
variants = [q["rewritten"]] + PHRASINGS[q["rewritten"]]
return rrf([search(v, k=5) for v in variants])[:10]
evaluate(multi, "three phrasings, fused", 2)
# three phrasings, fused 1.00 1.00 2Searching with a hypothetical answer#
A question and a passage that answers it do not look alike. Ask a model to write the answer it would expect, and search with that instead, and you are comparing a passage to a passage. It works when the model's guess is in the right neighbourhood and misfires when it is not, and the misfire is worth seeing rather than reading about.
HYDE = {
"How do refunds work on the team plan?":
"Refunds on team plans are pro-rated and issued within 30 days of the request.",
"What is the rate limit on the paid tier?":
"The paid tier allows several thousand requests per minute.",
"Is single sign-on available on team plans?":
"Single sign-on is available on team plans and can be configured in settings.",
"What happens if I add a seat to a team plan mid-month?":
"A seat added mid-month is charged pro-rata for the remainder of the period.",
"How long is data kept after an account is closed?":
"Data is retained for a period after closure and then permanently deleted.",
}
evaluate(lambda q: search(HYDE[q["rewritten"]], k=10), "hypothetical answer", 2)
for q in QUERIES[:1]:
print(" hyde query:", HYDE[q["rewritten"]])
print(" returns :", search(HYDE[q["rewritten"]]), "wanted", q["relevant"])
# hypothetical answer 1.00 0.77 2
# hyde query: Refunds on team plans are pro-rated and issued within 30 days of the request.
# returns : ['c1', 'c2', 'c3'] wanted {'c3'}Pulling the filter out of the question#
Some of what a question says is not a topic at all but a condition: a plan name, a product, a region, a date range. An embedding has to spend similarity on those words along with everything else, and it is bad at them, because one plan name looks much like another. Extract them and apply them as a metadata filter, and the search runs over the rows that could be right rather than over everything.
import re
PLANS = ["monthly", "annual", "team", "free", "paid", "enterprise"]
def extract_filter(question):
found = [p for p in PLANS if re.search(rf"\b{p}\b", question.lower())]
return found[0] if len(found) == 1 else None
question = "Is a refund issued in full on the annual plan?"
plan = extract_filter(question)
allowed = {i for i, (_, p) in CHUNKS.items() if p == plan}
print("filter:", plan, "candidates:", len(allowed), "of", len(CHUNKS))
print("no filter:", search(question))
print("filtered :", search(question, allowed=allowed))
# filter: annual candidates: 1 of 14
# no filter: ['c1', 'c2', 'c8']
# filtered : ['c2']Two rules keep this from causing its own outage. Extract a filter only when it is unambiguous — the code above returns nothing when two plan names appear, because filtering on the wrong one is worse than not filtering. And never let an extracted filter empty the result set silently: if the filtered search returns nothing, fall back to the unfiltered one and say so in the trace, or a typo in a product name turns into a confident 'I could not find anything'.
Breaking a question into parts#
A question with two subjects has no good single embedding, because the vector lands between them and is close to neither. Splitting it, searching each part and fusing is the fix, and it is worth doing only for the questions that need it — a classifier or a short model call decides, and most turns are not compound.
compound = "compare refunds on the monthly plan with the rate limit on the free tier"
parts = ["refunds on the monthly plan", "the rate limit on the free tier"]
want = {"c1", "c6"}
print("one search :", search(compound), "recall", recall_at_k(search(compound), want))
merged = rrf([search(p, k=3) for p in parts])
print("decomposed :", merged[:3], "recall", recall_at_k(merged, want))
# one search : ['c1', 'c2', 'c13'] recall 0.5
# decomposed : ['c2', 'c1', 'c6'] recall 1.0Put the four rows together and the shape of the decision is clear. The standalone rewrite is the one that pays on a chat interface, and it pays enormously. Multiple phrasings cost a call and returned nothing here. A hypothetical answer kept recall and damaged ranking. Decomposition is worth having for the questions that need it and nothing for the ones that do not. The measurement is cheap — fourteen chunks and five questions took an afternoon to write — and it is the only thing that distinguishes those four outcomes, which is why stacking all of them from a blog post leaves you slower and unable to say what helped.
Do not rewrite what is already standalone
The rewrite adds a model call and a few hundred milliseconds to every turn, including the first one, which by definition has no history to fold in. A cheap test skips most of that: if the turn is the first in the conversation, or it is long enough and contains a noun the history does not supply, send it as it is. Keep the check conservative, because a turn wrongly judged standalone is retrieved on the wrong query and the user sees a confidently irrelevant answer, while a turn wrongly rewritten costs only the call.
Where people go wrong
Adding every query technique from a blog post at once. Latency triples and you cannot tell which one helped.
If this comes up in an interview
Your RAG chatbot answers the first question well and follow-ups badly. Why, and what do you do?
Follow-ups depend on earlier turns, so embedding the raw message retrieves the wrong things. Rewrite each turn into a standalone question using the history before retrieving, and measure the change on a set of real follow-up questions.
Then they push further
The rewrite adds 300ms. Is it worth it?
Measure it. If recall on follow-ups rises substantially, usually yes, and a small fast model keeps the delay down. Skip the rewrite when the message is already self-contained, which a cheap check can detect.
What gets marked up. Marked up for spotting that the query lacks context and for measuring the fix rather than stacking techniques.
What is MRR and when would you use it rather than recall@k?
Mean reciprocal rank averages one over the position of the first relevant result. It rewards getting a relevant result to the very top, which matters when only the first few results are used. Recall@k measures whether the relevant results made it into the set at all.
When would you search with a hypothetical answer rather than the question, and what is the risk?
When questions and passages are written differently enough that a question embedding sits far from the passage that answers it. You ask a model for the answer it would expect and search with that, so you are comparing a passage to a passage. The risk is that the guess is specific and wrong: an invented detail becomes part of the query and pulls the search towards documents containing that detail rather than the correct one. Measure recall and MRR separately, because the usual damage is to the ranking rather than to whether the right chunk is retrieved at all.
Then they push further
How would you make it safer?
Search with the question and the hypothetical answer and fuse the rankings, so an off-target guess cannot dominate. Keep the hypothetical short and generic. And measure on questions where your corpus disagrees with common knowledge, since that is where the model's guess is confidently wrong.
What gets marked up. Marked up for naming the failure concretely, and for knowing the technique trades ranking for vocabulary match rather than being a free improvement.
How do you decide which query techniques to keep?
Build a small labelled set: twenty or so real turns from a chat log with the chunks that answer them. Measure recall@k and MRR for each technique added one at a time, and record the extra model calls and latency beside them. Keep the ones that move a number you care about. On most chat corpora the standalone rewrite moves recall a long way and the rest move it very little, and you cannot tell which case you are in without the measurement.
Build this: Collect twenty real follow-up questions from a chat log with their answering chunks. Measure recall@5 and MRR for raw messages, then with standalone rewriting, then with one more technique of your choice. Keep only what earns its latency.
Watch it explained
This is one section of Reranking & hybrid search, 10 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 25 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.