Making it actually work
By the end: You can diagnose a failing RAG system and know which stage to fix.
- Measuring retrieval separately from generation
- Query rewriting
- Handling the no-answer case
- Keeping the index fresh
New to retrieval-augmented generation? RAG stands for retrieval-augmented generation, which describes it badly. The idea is plain: before asking the model a question, go and find the handful of documents that bear on it and put them in the request, so the answer comes from material in front of the model rather than from whatever it happens to remember. It is the difference between an exam sat from memory and an open-book one where you fetch the right pages first. Why it matters
You can build one of these now. This section is about the harder part, which is fixing one that answers badly. There are only two places a bad answer can come from and they take completely different repairs, so the work starts by separating them and measuring each on its own. The rest covers rewriting queries before they are searched, letting the system say it does not know, and keeping an index from going stale.
Measure the stages separately#
A RAG system can fail at retrieval or at generation, and the fixes have nothing in common. Measure them apart or you will guess.
Retrieval: was the correct chunk in the top k?
a yes or no per question, over a fixed set
Generation: given the correct chunk, was the answer right?
feed the right chunk deliberately and score the answerThat is easier to agree with than to act on, so here it is on a corpus small enough to read: a twelve-section support handbook, thirteen questions with the chunk and the sentence that answers each one written down in advance, and a retriever and a generator that both run offline. The retriever is TF-IDF over character n-grams, which matches refund against refunds and stands in for an embedding model without needing one. The generator quotes the sentence in the context sharing most words with the question. Neither is as good as the real thing, and both fail in the directions the real things fail in, which is what the measurements are for.
import re
from sklearn.feature_extraction.text import TfidfVectorizer
HANDBOOK = """# Refunds
An approved refund is returned to the original payment method within 5 working days.
International refunds take up to 10 working days because of bank clearing.
# Returns
An item may be returned within 30 days of delivery if it is unused.
Sale items are returned under the same window as everything else.
# Delivery
Standard delivery arrives in 2 to 4 working days.
Express delivery is next day if the order is placed before 3pm.
# Addresses
We do not deliver to PO boxes.
A delivery address can be changed at any time before the order is dispatched.
# Accounts
The email address on an account is changed from the account settings page.
Deleting an account removes its order history after 30 days.
# Passwords
A password reset link is valid for 60 minutes.
Three failed sign-in attempts lock the account for 15 minutes.
# Payment
We accept cards and direct debit.
We do not accept cheques or bank transfers.
# Invoices
Invoices are issued on the first of each month.
A VAT receipt can be downloaded from the billing page.
# Damage
Damage in transit must be reported within 48 hours with a photograph.
A replacement is dispatched in the same week the report is accepted.
# Gift cards
A gift card is valid for 12 months from purchase.
Gift cards cannot be exchanged for cash.
# Subscriptions
A subscription can be cancelled at any time.
Cancellation takes effect at the end of the period already paid for.
# Contact
Support is open from 9am to 6pm on weekdays.
Live chat is on the help page.
"""
CHUNKS = []
for block in HANDBOOK.strip().split("\n\n"):
lines = block.strip().split("\n")
CHUNKS.append({"id": "c%d" % len(CHUNKS),
"heading": lines[0].lstrip("# ").strip(),
"text": " ".join(lines[1:]).strip()})
BY_ID = {c["id"]: c for c in CHUNKS}
# Character n-grams, so refund matches refunds. A stand-in for an embedding
# model: no network, same numbers on every run.
vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(4, 5), sublinear_tf=True)
INDEX = vec.fit_transform([c["heading"] + ". " + c["text"] for c in CHUNKS])
def retrieve(question, k=3):
scores = (INDEX @ vec.transform([question]).T).toarray().ravel()
return [(CHUNKS[i], float(scores[i])) for i in scores.argsort()[::-1][:k]]
STOP = set("a an and are be by can do does for from get have how i if in is it "
"many me my of on or the to what when where will you your".split())
def keys(text):
return {w[:4] for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}
def sentences(c):
return [s.strip() + "." for s in c["text"].split(".") if s.strip()]
def generate(question, chunks):
"""Quote the sentence sharing most words with the question. Ties go to the
first one seen, which is why the order of the context matters below."""
want, best, high, cite = keys(question), None, 0, None
for c in chunks:
for s in sentences(c):
n = len(want & keys(s))
if n > high:
best, high, cite = s, n, c["id"]
return ("I do not know.", None) if best is None else (best, cite)
print(len(CHUNKS), "chunks,", sum(len(c["text"]) for c in CHUNKS), "characters")
# 12 chunks, 1352 characters# question, the chunk that should be found, which of its two sentences answers it
EVAL = [
("how long does a refund to my card take", "c0", 0),
("how long does a refund to an overseas bank take", "c0", 1),
("how long do I have to return an item", "c1", 0),
("can I return a discounted item", "c1", 1),
("when will standard delivery arrive", "c2", 0),
("how late can I order for next day delivery", "c2", 1),
("how long is a password reset link valid", "c5", 0),
("what happens after several failed logins", "c5", 1),
("how long is a gift card good for", "c9", 0),
("can I exchange a gift card for money", "c9", 1),
("if I cancel my subscription today when does it stop", "c10", 1),
("my parcel turned up smashed", "c8", 0),
("when can I speak to a human", "c11", 0),
]
def gold(chunk_id, i):
return sentences(BY_ID[chunk_id])[i]
for k in (1, 3, 5):
found = sum(g in [c["id"] for c, _ in retrieve(q, k)] for q, g, _ in EVAL)
print("retrieval recall@%d %.2f" % (k, found / len(EVAL)))
given = sum(generate(q, [BY_ID[g]])[0] == gold(g, i) for q, g, i in EVAL)
print("answers right given the gold chunk %.2f" % (given / len(EVAL)))
end = sum(generate(q, [c for c, _ in retrieve(q, 3)])[0] == gold(g, i) for q, g, i in EVAL)
print("answers right end to end %.2f" % (end / len(EVAL)))
for q, g, i in EVAL:
got, _ = generate(q, [BY_ID[g]])
if got != gold(g, i):
print(" from the right chunk:", q)
print(" said:", got)
# retrieval recall@1 0.85
# retrieval recall@3 0.92
# retrieval recall@5 0.92
# answers right given the gold chunk 0.62
# answers right end to end 0.62
# from the right chunk: how long does a refund to my card take
# said: International refunds take up to 10 working days because of bank clearing.
# from the right chunk: can I return a discounted item
# said: An item may be returned within 30 days of delivery if it is unused.
# from the right chunk: if I cancel my subscription today when does it stop
# said: A subscription can be cancelled at any time.
# from the right chunk: my parcel turned up smashed
# said: I do not know.
# from the right chunk: when can I speak to a human
# said: I do not know.Read the three failures underneath. Asked how long a refund to a card takes, it quotes the sentence about international payments, because the word take appears in that sentence and not in the right one. Asked about a discounted item, it quotes the general returns rule rather than the sale-items sentence, because the document says sale and the question says discounted. Asked when a cancellation takes effect, it quotes the sentence saying a subscription can be cancelled at any time, which is true, adjacent and not an answer. Every one of these is the same shape: the right chunk, the wrong sentence in it. A single end-to-end score of 0.62 would have told you nothing about which of the two stages to work on, and the pair 0.92 and 0.62 tells you immediately.
Nothing relevant came back, and an answer arrived anyway#
The failure that costs the most trust is the one where retrieval returns nothing useful and the system answers from what the model already knew. Models do not experience an empty retrieval as missing information; they experience it as a question, and they are fluent. Here that behaviour is made explicit as a lookup table the stub falls back on when nothing in the context matches, so it can be switched off and measured rather than argued about.
# What the stub "learnt in training": plausible, unsourced, mostly wrong.
PRIOR = {
"student": "Yes, students get 10% off with a valid card.",
"app": "Yes, there is an app for iOS and Android.",
"collect": "Yes, you can collect from any store.",
"phone": "Support is on 0800 100 200.",
"vat": "The VAT number is GB123456789.",
}
UNANSWERABLE = ["do you offer a student discount",
"is there a mobile app",
"can I collect my order from a shop",
"what is your phone number",
"what is your VAT number"]
def ask(question, k=3, floor=0.0, use_prior=True):
hits = retrieve(question, k)
if hits[0][1] < floor:
return "I do not know."
text, cite = generate(question, [c for c, _ in hits])
if cite is None and use_prior:
for trigger, fallback in PRIOR.items():
if trigger in question.lower():
return fallback
return text
print("grounded only:")
for q in UNANSWERABLE:
print(" %-34s %s" % (q, ask(q, use_prior=False)))
print("allowed to fall back on what it knows:")
for q in UNANSWERABLE:
print(" %-34s %s" % (q, ask(q, use_prior=True)))
# grounded only:
# do you offer a student discount I do not know.
# is there a mobile app I do not know.
# can I collect my order from a shop A delivery address can be changed at any
# time before the order is dispatched.
# what is your phone number I do not know.
# what is your VAT number A VAT receipt can be downloaded from the
# billing page.
# allowed to fall back on what it knows:
# do you offer a student discount Yes, students get 10% off with a valid card.
# is there a mobile app Yes, there is an app for iOS and Android.
# can I collect my order from a shop A delivery address can be changed at any
# time before the order is dispatched.
# what is your phone number Support is on 0800 100 200.
# what is your VAT number A VAT receipt can be downloaded from the
# billing page.Those last two matter more than the invented ones. Forbidding the model to use outside knowledge does not stop a confident wrong answer, because the retriever handed it real text on a neighbouring subject and answering from that text is exactly what it was told to do. Grounding is not a defence against irrelevance. What catches these is the score, not the instruction.
A floor, and what the floor costs#
answerable = [retrieve(q, 1)[0][1] for q, _, _ in EVAL]
unanswerable = [retrieve(q, 1)[0][1] for q in UNANSWERABLE]
print("answerable min %.2f median %.2f max %.2f"
% (min(answerable), sorted(answerable)[len(answerable) // 2], max(answerable)))
print("unanswerable min %.2f median %.2f max %.2f"
% (min(unanswerable), sorted(unanswerable)[len(unanswerable) // 2], max(unanswerable)))
print("floor unanswerable let through answerable blocked")
for floor in (0.00, 0.15, 0.20, 0.30, 0.40):
print("%.2f %d of 5 %d of 13"
% (floor, sum(s >= floor for s in unanswerable), sum(s < floor for s in answerable)))
# answerable min 0.15 median 0.45 max 0.66
# unanswerable min 0.08 median 0.13 max 0.28
# floor unanswerable let through answerable blocked
# 0.00 5 of 5 0 of 13
# 0.15 2 of 5 0 of 13
# 0.20 1 of 5 2 of 13
# 0.30 0 of 5 3 of 13
# 0.40 0 of 5 5 of 13So the threshold is a product decision with a measured price, not a constant to copy from a tutorial. For a support bot, refusing a question the handbook covers is a smaller harm than inventing a VAT number, and the floor goes high. For a research assistant used by people who check everything, a wrong answer costs a minute and a refusal costs a session, and the floor goes low. What is not defensible is choosing 0.7 because it looked confident, on a score whose scale you never plotted.
Right, and unsupported#
Correctness and groundedness are different properties and only one of them is under your control. An answer can be true and absent from the retrieved text, which means the system got lucky rather than working, and the same code will invent the next one. Measuring only accuracy hides that completely.
def supported(text, chunks):
"""Is every content word of the answer present in the retrieved text?"""
context = set()
for c in chunks:
context |= keys(c["text"])
return keys(text) <= context
# True where the stub's answer happens to be factually right anyway.
TRUTH = {"do you offer a student discount": False,
"is there a mobile app": False,
"can I collect my order from a shop": False,
"what is your phone number": True,
"what is your VAT number": False}
grid = {}
for q in UNANSWERABLE:
out = ask(q, use_prior=True)
cell = (TRUTH[q], supported(out, [c for c, _ in retrieve(q, 3)]))
grid[cell] = grid.get(cell, 0) + 1
print("right=%-5s supported=%-5s %s" % (cell[0], cell[1], out))
print()
for right in (True, False):
for sup in (True, False):
print("right=%-5s supported=%-5s %d" % (right, sup, grid.get((right, sup), 0)))
# right=False supported=False Yes, students get 10% off with a valid card.
# right=False supported=False Yes, there is an app for iOS and Android.
# right=False supported=True A delivery address can be changed at any time
# before the order is dispatched.
# right=True supported=False Support is on 0800 100 200.
# right=False supported=True A VAT receipt can be downloaded from the billing page.
#
# right=True supported=True 0
# right=True supported=False 1
# right=False supported=True 2
# right=False supported=False 2In a real evaluation the support check is usually another model asked whether each claim follows from the retrieved text, which is a judge and needs validating like any other judge. The shape of the measurement does not change: one score for whether the answer is right, a second for whether the retrieved text supports it, and a rule that an unsupported answer is a defect even when it is correct. Teams that report a single quality number ship the phone-number row and find out from a customer.
The order the context is assembled in#
Retrieval returns chunks with scores and the prompt is a flat string, so something has to decide what goes in and in what order. That decision is usually made by accident, in a line that sorts the chunks for readability and then trims whatever does not fit the budget. Those two steps in that order throw away the best evidence.
def fit(chunks, budget=400):
"""Keep chunks in the order given, until the budget runs out."""
kept, used = [], 0
for c in chunks:
if used + len(c["text"]) <= budget:
kept.append(c)
used += len(c["text"])
return kept
by_score = lambda hits: [c for c, _ in hits]
in_document_order = lambda chunks: sorted(chunks, key=lambda c: int(c["id"][1:]))
def run(select, present, label):
kept_gold = right = 0
for q, g, i in EVAL:
context = present(fit(select(retrieve(q, 6))))
kept_gold += g in [c["id"] for c in context]
right += generate(q, context)[0] == gold(g, i)
print("%-46s gold kept %2d/13 right %2d/13" % (label, kept_gold, right))
run(by_score, lambda cs: cs, "trim in score order, best chunk first")
run(lambda h: in_document_order(by_score(h)), lambda cs: cs, "trim in document order")
run(by_score, lambda cs: cs[::-1], "trim in score order, best chunk last")
run(by_score, in_document_order, "trim in score order, present in document order")
# trim in score order, best chunk first gold kept 12/13 right 8/13
# trim in document order gold kept 9/13 right 6/13
# trim in score order, best chunk last gold kept 12/13 right 7/13
# trim in score order, present in document order gold kept 12/13 right 7/13The rule that falls out is short. Choose what fits by score, then choose the order for the reader's sake, and never let a sort run between the two. The remaining difference in that table, between 8 and 7 answers, comes from where the best chunk sits once everything has been selected, and the stub is a poor model of that: it breaks ties by position, which is a crude version of a real effect. Models are measurably better at using material at the beginning and the end of a long context than material buried in the middle, an effect documented well enough to plan around. With a handful of chunks it does not matter. With forty it does, and the cheap defence is to send fewer chunks rather than to guess at an ordering.
Query rewriting#
Users do not write good search queries. They write 'what about the other thing' on turn four. Rewriting the query using the conversation before searching often helps more than any tuning of the index.
History: "How long do refunds take?"
"Five working days."
User: "and for international?"
Rewritten: "How long do international refunds take?"The no-answer case#
The failure that destroys trust
A RAG system confidently answering a question its documents do not cover is worse than one that says it does not know. Users discover the invention once and stop trusting every answer after it.
- Set a relevance threshold. If the best chunk scores below it, do not answer from retrieval at all.
- Instruct the model explicitly that declining is a correct answer when the sources do not cover the question.
- Put no-answer cases in your eval set. They are the ones nobody tests and the ones that damage trust.
Freshness#
Documents change. An index built once and never updated slowly becomes a confident source of outdated answers, which is a particularly hard failure to notice because nothing errors.
- Re-index changed documents rather than rebuilding everything, keyed on a content hash.
- Store a last-updated date on each chunk and show it with the citation.
- Delete chunks for documents that were removed, or they will keep being cited.
Where people go wrong
Evaluating the whole pipeline with one number. It tells you something is wrong and nothing about where, so every fix is a guess.
If this comes up in an interview
Your RAG system gives a wrong answer. How do you find out why?
Separate the stages. Check whether the correct chunk was retrieved. If it was not, the problem is chunking, embedding or ranking. If it was, feed it deliberately and check the answer, which isolates a prompt or model problem.
How do you handle a question your documents cannot answer?
Threshold on retrieval score so a weak best match does not proceed, instruct the model that declining is acceptable, and include no-answer cases in the eval set. A confident wrong answer costs more trust than a refusal.
Then they push further
How do you measure how well it declines?
Put unanswerable questions in the eval set on purpose and measure how often it answers them anyway, alongside how often it wrongly declines questions it could answer. Improving one usually worsens the other, so track both.
What gets marked up. Marked up for treating 'I do not know' as a correct answer with its own test cases, and for tracking wrong refusals too.
What is query rewriting and why does it matter?
Rewriting the user's question into a standalone search query using the conversation context. Follow-up questions like 'and for international?' are meaningless to a search index on their own, and rewriting often improves retrieval more than index tuning.
How do you keep a RAG index current?
Re-index on change rather than rebuilding, keyed by content hash. Delete chunks for removed documents. Store and display a last-updated date, because a stale answer is indistinguishable from a correct one to the user.
Build this: Build separate retrieval and generation evals over thirty questions, including five the documents cannot answer. Report recall at 5, generation accuracy given correct context, and how often the system wrongly answers a no-answer question.
Watch it explained
This is one section of Retrieval-augmented generation, 8 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 24 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.