Asked about Where embedding search fails
Your RAG system returns wrong answers. How do you debug it?
An answer that works
Check recall first: was the correct document in the retrieved set at all? If not, the problem is chunking or embedding. If it was retrieved but ranked low, the problem is ranking, and reranking or hybrid search will help. Only if retrieval was correct is it a generation problem.
Then they push further
The right chunk was retrieved but the answer is still wrong. Now what?
The problem is in generation. Check that the chunk was in the context the model actually saw and not trimmed away, whether it was buried among irrelevant chunks, and whether the prompt tells the model to rely on the context. Then read the answer against the chunk to see if it misread a number or blended two sources.
The right chunk was never retrieved. What next?
Find out why. Try the question as a keyword search to see if the words match at all, check how that document was parsed and chunked, and check whether the question uses different words from the document. Hybrid search, better chunking and query rewriting each fix a different one of these.
How do you stop it coming back?
Add the question to the retrieval eval with the chunk it should find, and to the answer eval with the expected answer. Measure retrieval separately from answer quality so you know which stage regressed.
What gets marked up
Strong answers split the problem into retrieval and generation and check which failed before changing anything. Jumping straight to a new embedding model or a bigger model is marked down.
Reading an answer is not being able to give one. The section that teaches this is Where embedding search fails, 3 hours of Embeddings.
Learn this properly