Where embedding search fails
By the end: You can predict which queries will fail and fix them.
- Topical similarity is not relevance
- Exact terms that embeddings lose
- Hybrid search
- Reranking
New to embeddings? An embedding turns a piece of text into a long list of numbers — a few hundred or a few thousand of them. Think of those numbers as coordinates, the way latitude and longitude place a town on a map. Text about similar things lands in a similar place, so "how do I reset my password" ends up near "I forgot my login" even though they share almost no words. Why it matters
Vector search is easy to get working and easy to trust too far. This section is about the queries it handles badly, which is a wider set than the demos suggest: exact identifiers, proper names, negations, and anything where the passage you need is worded nothing like the question. You will learn to predict those in advance, and the two standard repairs, running a keyword search alongside the vector search and reranking the combined results.
Similar is not the same as useful#
This is the most important thing to understand about retrieval. Embeddings find text about the same subject. They do not find text that answers your question. Those overlap often enough to be encouraging and diverge often enough to be a problem.
Query: "Does the warranty cover water damage?"
Retrieved, scoring highly, all about warranties:
"Our warranty lasts 24 months."
"Warranty claims require a receipt."
"The warranty covers manufacturing defects."
None of them answer the question.Exact terms disappear#
Embeddings compress meaning, and in doing so they lose precision on exact strings. A product code, an error number, a surname, a rare technical term: these are exactly what a user searches for, and exactly what embeddings are worst at.
Query: "error E4471"
Embedding search: returns general text about errors
Keyword search: returns the document containing E4471Use both#
Hybrid search runs a keyword search and a vector search and combines the results. Keyword search catches exact terms, vector search catches paraphrase. Together they cover each other's blind spots, and this is usually the single largest quality improvement available to a mediocre retrieval system.
def hybrid(query, k=10):
keyword = bm25_search(query, k=k)
vector = vector_search(query, k=k)
# Reciprocal rank fusion: score by position, not by raw score,
# because the two systems' scores are not on the same scale.
scores = {}
for rank, doc in enumerate(keyword):
scores[doc.id] = scores.get(doc.id, 0) + 1 / (60 + rank)
for rank, doc in enumerate(vector):
scores[doc.id] = scores.get(doc.id, 0) + 1 / (60 + rank)
return sorted(scores, key=scores.get, reverse=True)[:k]Rerank the shortlist#
A reranker is a slower, more accurate model that scores each candidate against the query directly, rather than comparing two independently computed vectors. It is too slow to run over a whole corpus and ideal for reordering twenty candidates down to five.
Retrieve broadly 50 candidates fast, high recall
Rerank top 20 slower, high precision
Send to the model top 5 expensive per tokenDiagnose recall before precision
If the right document was never retrieved, no amount of reranking helps. Check first whether the answer is in your top fifty. If it is, you have a ranking problem. If it is not, you have a chunking or an embedding problem, and they need different fixes.
Where people go wrong
Tuning the prompt when retrieval is the problem. If the right passage never reached the model, no prompt can fix it, and teams lose weeks this way.
If this comes up in an interview
Why does embedding search sometimes return irrelevant results?
It measures topical similarity, not whether a passage answers the question. Text about the right subject scores highly even when it contains nothing useful. It also loses exact terms like product codes, which is why hybrid search exists.
What is hybrid search?
Running keyword and vector search together and fusing the results, usually by rank rather than raw score since the two scoring scales are incomparable. Keyword catches exact terms, vector catches paraphrase.
What does a reranker do?
It scores each candidate against the query directly rather than comparing pre-computed vectors, which is more accurate and much slower. You use it to reorder a shortlist, never to search a whole corpus.
Your RAG system returns wrong answers. How do you debug it?
Check recall first: was the correct document in the retrieved set at all? If not, the problem is chunking or embedding. If it was retrieved but ranked low, the problem is ranking, and reranking or hybrid search will help. Only if retrieval was correct is it a generation problem.
Then they push further
The right chunk was retrieved but the answer is still wrong. Now what?
The problem is in generation. Check that the chunk was in the context the model actually saw and not trimmed away, whether it was buried among irrelevant chunks, and whether the prompt tells the model to rely on the context. Then read the answer against the chunk to see if it misread a number or blended two sources.
The right chunk was never retrieved. What next?
Find out why. Try the question as a keyword search to see if the words match at all, check how that document was parsed and chunked, and check whether the question uses different words from the document. Hybrid search, better chunking and query rewriting each fix a different one of these.
How do you stop it coming back?
Add the question to the retrieval eval with the chunk it should find, and to the answer eval with the expected answer. Measure retrieval separately from answer quality so you know which stage regressed.
What gets marked up. Strong answers split the problem into retrieval and generation and check which failed before changing anything. Jumping straight to a new embedding model or a bigger model is marked down.
Build this: Build a search over fifty documents. Write five queries that embedding search handles well and five it handles badly, including one with an exact product code. Then add keyword search and measure the improvement.
Watch it explained
This is one section of Embeddings, 5 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 21 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.