DataCore8h

Retrieval-augmented generation

Fetch the relevant text, then answer from it.

RAG stands for retrieval-augmented generation, which describes it badly. The idea is plain: before asking the model a question, go and find the handful of documents that bear on it and put them in the request, so the answer comes from material in front of the model rather than from whatever it happens to remember. It is the difference between an exam sat from memory and an open-book one where you fetch the right pages first.

A model knows nothing about your company's documents, and its training stopped on a fixed date, so asking it about either produces fluent, confident and wrong answers. Putting the source text in the request also means the answer can point back at where it came from, which is what makes it checkable by someone who cares whether it is true.

It is the top of the retrieval stack and only makes sense once the pieces under it do: chunking cuts documents into passages, embeddings turn them into coordinates, a vector database finds the nearest ones, and reranking puts the best first. It competes with simply sending everything in a large context window, and wins on cost and on accuracy. Offering retrieval as a tool the agent chooses to call, rather than a fixed stage before every answer, is usually the stronger design.

Fetch the evidence first and answer from it, because the model's memory is not the source of truth.

In more detailRAG puts source material into the request so the model answers from evidence rather than memory. A working system needs chunking that respects document structure, hybrid keyword-and-vector retrieval, reranking, and citations back to the source. Treating retrieval as a tool the agent chooses to call, rather than a fixed pipeline stage, is usually the stronger design.

Learn first

Embeddings Tool calling Chunking & document processing

Section 1 of 2, 4h

The RAG pipeline

By the end: You can build a retrieval system end to end and explain each stage.

What RAG is and what problem it solves
The five stages
Chunking, which is where most quality is lost
Citations

Retrieval-augmented generation is the standard way to make a model answer from your material rather than from whatever it absorbed in training. Find the relevant passages, put them in the prompt, ask the question there. This section is the whole thing end to end, from splitting documents through to citing where an answer came from, and it gives most of its attention to chunking, which is dull and decides more of the final quality than the choice of model does.

The idea#

Retrieval, and the step people skip

1 / 6

Documents are split into chunks, and each chunk is embedded once, in advance.Nearby points mean similar text. The index is built before any question arrives.

A model knows what was in its training data. It does not know your documents, and it will not admit that gracefully. RAG fixes this by finding the relevant text and putting it in the prompt, so the model answers from evidence in front of it rather than from memory.

Question  ->  retrieve relevant chunks  ->  put them in the prompt
          ->  model answers using them  ->  answer plus citations

The stages#

1.Ingest: parse documents into clean text. PDFs, HTML and tables all fight you here.
2.Chunk: split into pieces small enough to retrieve precisely and large enough to make sense alone.
3.Embed and index: convert chunks to vectors and store them alongside the original text and metadata.
4.Retrieve: hybrid search, then rerank, to get a handful of good chunks.
5.Generate: put those chunks in the prompt and ask the model to answer using only them.

The whole thing, running#

Each of those stages is a function you can write yourself, and the version that fits on one screen is worth building before adopting anything that hides it. What follows is a complete pipeline: four documents, a chunker, an index, retrieval, prompt assembly and a generator, none of which touches a network. The generator is a stub that quotes whichever sentence in the retrieved text shares most words with the question, so the pipeline runs offline and gives the same answer every time. Replacing it with a model call changes one function and leaves the other four alone, which is the useful thing to see.

import re
from sklearn.feature_extraction.text import TfidfVectorizer

HANDBOOK = """# Refunds
An approved refund is returned to the original payment method within 5 working days.
International refunds take up to 10 working days because of bank clearing.

# Returns
An item may be returned within 30 days of delivery if it is unused.
Sale items are returned under the same window as everything else.

# Delivery
Standard delivery arrives in 2 to 4 working days.
Express delivery is next day if the order is placed before 3pm.

# Accounts
The email address on an account is changed from the account settings page.
Deleting an account removes its order history after 30 days.
"""

def chunk(document):
    """One chunk per heading, with the heading kept on the text."""
    out = []
    for block in document.strip().split("\n\n"):
        lines = block.strip().split("\n")
        out.append({"id": "c%d" % len(out),
                    "heading": lines[0].lstrip("# ").strip(),
                    "text": " ".join(lines[1:]).strip()})
    return out

CHUNKS = chunk(HANDBOOK)
for c in CHUNKS:
    print(c["id"], c["heading"], len(c["text"]), "chars")

vec = TfidfVectorizer(stop_words="english")
INDEX = vec.fit_transform([c["heading"] + ". " + c["text"] for c in CHUNKS])
print("index:", INDEX.shape[0], "chunks by", INDEX.shape[1], "terms")

# c0 Refunds 159 chars
# c1 Returns 133 chars
# c2 Delivery 113 chars
# c3 Accounts 135 chars
# index: 4 chunks by 38 terms

Stage one and two. In production the index is an embedding model and a vector store; here it is TF-IDF over four chunks, which keeps the pipeline visible instead of the infrastructure. The heading is glued to the text before indexing, so the word Refunds is searchable even though the sentences under it say refund.

Retrieval is a similarity and a sort. Assembly is string formatting. Neither is clever, and both are where systems go wrong, so it is worth watching what they actually produce rather than trusting a library to produce it.

def retrieve(question, k=2):
    scores = (INDEX @ vec.transform([question]).T).toarray().ravel()
    return [(CHUNKS[i], float(scores[i])) for i in scores.argsort()[::-1][:k]]

for c, s in retrieve("how long does a refund take", k=3):
    print("%.3f  %s  %s" % (s, c["id"], c["heading"]))

# 0.241  c0  Refunds
# 0.000  c3  Accounts     <- asked for three, so three came back
# 0.000  c2  Delivery

def assemble(question, hits):
    lines = ["Answer using only the sources below.",
             "Cite the source id in brackets after each claim.",
             "If the sources do not answer the question, say you do not know.", ""]
    lines += ["[%s] %s: %s" % (c["id"], c["heading"], c["text"]) for c, _ in hits]
    lines += ["", "Question: " + question]
    return "\n".join(lines)

print(assemble("how long does a refund take", retrieve("how long does a refund take")))

# Answer using only the sources below.
# Cite the source id in brackets after each claim.
# If the sources do not answer the question, say you do not know.
#
# [c0] Refunds: An approved refund is returned to the original payment method
#      within 5 working days. International refunds take up to 10 working days
#      because of bank clearing.
# [c3] Accounts: The email address on an account is changed from the account
#      settings page. Deleting an account removes its order history after 30 days.
#
# Question: how long does a refund take

The second retrieved chunk scores 0.000 and is about account deletion. A fixed k always returns k results, so an irrelevant chunk is in the prompt whenever fewer than k good ones exist. That is not a bug in the retriever, it is the shape of the interface, and it is why a score floor appears in the next section.

The last stage is the model, and a stub with known behaviour teaches more here than an API key would. This one extracts rather than composes, which is cruder than a real model but fails in a recognisable direction, and it declines when retrieval came back with nothing above a floor.

STOP = set("a an and are be by can do does for from get have how i if in is it "
           "many me my of on or the to what when where will you your".split())

def keys(text):
    """Four-letter prefixes, so refund and refunds match."""
    return {w[:4] for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}

def generate(question, hits):
    """Stands in for the model: quote the sentence sharing most words with the
    question, and cite the chunk it came from. Deterministic, offline."""
    want, best, high, cite = keys(question), None, 0, None
    for c, _ in hits:
        for sentence in [s.strip() + "." for s in c["text"].split(".") if s.strip()]:
            overlap = len(want & keys(sentence))
            if overlap > high:
                best, high, cite = sentence, overlap, c["id"]
    if best is None:
        return "I do not know."
    return "%s [%s]" % (best, cite)

def ask(question, k=2, floor=0.15):
    hits = retrieve(question, k)
    if hits[0][1] < floor:
        return "I do not know. Best chunk scored %.3f, under the floor." % hits[0][1]
    return generate(question, hits)

for q in ["how many days do I have to return an item",
          "how long do international refunds take",
          "how long does a refund take",
          "do you ship to Ireland"]:
    print(q)
    print("   ", ask(q))

# how many days do I have to return an item
#     An item may be returned within 30 days of delivery if it is unused. [c1]
# how long do international refunds take
#     International refunds take up to 10 working days because of bank clearing. [c0]
# how long does a refund take
#     International refunds take up to 10 working days because of bank clearing. [c0]
# do you ship to Ireland
#     I do not know. Best chunk scored 0.000, under the floor.

Four questions, three behaviours. The first two are right. The third is wrong from the right chunk: retrieval found Refunds, and the word take in the question matched the sentence about international payments rather than the one about five working days. The fourth declines, because nothing scored above the floor.

That third answer is worth staring at, because it is the failure this topic is mostly about. Retrieval did its job perfectly. The correct sentence was in the prompt. The answer is still wrong, and no change to the index, the embedding model or the chunk size would have prevented it. A system measured by a single end-to-end score would record that as a retrieval problem and send you to tune the wrong half. The next section splits the measurement so it cannot.

Chunking is where quality is won or lost#

Splitting every 500 characters is the default advice and it is bad. It cuts sentences in half, separates a heading from the text under it, and splits tables down the middle. The chunk then reads as nonsense and cannot be retrieved sensibly.

Bad, split at a fixed character count:

  chunk 41: "...and the refund period is"
  chunk 42: "5 working days from receipt..."

Neither chunk answers "how long do refunds take".

Split on structure instead. Paragraphs, sections, headings. Keep the heading with its content, and attach the document title to every chunk so a fragment still carries its context.

DOC = {"title": "Support handbook", "url": "https://example.com/handbook"}

enriched = [{"text": c["text"],
             "title": DOC["title"],            # so the chunk is self-describing
             "heading": c["heading"],
             "source": DOC["url"],
             "position": position}
            for position, c in enumerate(CHUNKS)]

# What gets embedded includes the context, not just the raw text.
def embedding_text(chunk):
    return "%s > %s\n\n%s" % (chunk["title"], chunk["heading"], chunk["text"])

print(embedding_text(enriched[0]))

# Support handbook > Refunds
#
# An approved refund is returned to the original payment method within 5 working
# days. International refunds take up to 10 working days because of bank clearing.

Embedding the heading path alongside the text substantially improves retrieval on structured documents.

Overlap slightly

Let consecutive chunks share a sentence or two. It costs a little storage and stops an answer being lost because it happened to straddle a boundary.

Citations#

Give each chunk an identifier and ask the model to cite the ones it used. This gives the user a way to verify, gives you a way to debug, and measurably reduces invention, because a model asked to cite is less inclined to assert something it cannot point at.

Answer using only the sources below. After each claim, cite the
source id in square brackets. If the sources do not contain the
answer, say so and cite nothing.

[doc_12] Refunds are processed within 5 working days.
[doc_37] Returns are accepted within 30 days of delivery.

Citations also change what you can debug. An answer with a source id can be checked against that source mechanically, which is the difference between knowing an answer is wrong and knowing which stage made it wrong. An answer without one can only be argued about.

When RAG is the wrong tool#

Retrieval is the default answer to so many questions that it gets applied to problems it does not fit, and the resulting system is slower, more expensive and worse than the thing it replaced. Four cases come up repeatedly.

The corpus is small enough to send. Twenty pages of policy is a few thousand tokens. Putting all of it in the prompt has perfect recall, no index to maintain and no retrieval stage to debug, and prompt caching makes the repeated cost small. Retrieval earns its complexity when the corpus cannot fit, not when it is merely large-ish.
The question is not a lookup. Aggregates, comparisons and counts are database queries wearing a question mark. How many orders were refunded last month is answered by SQL, and a retriever that returns the five most similar order records will answer it confidently and wrongly. Give the model the query tool instead.
The problem is behaviour, not knowledge. A consistent output format, a house tone, a classification the model keeps getting wrong: none of these are fixed by putting documents in the prompt. Prompting, examples or fine-tuning address those, and retrieval adds latency without touching the cause.
The answer changes per user and lives in a system of record. Account balance, order status, entitlement. These want an API call with the user's identity attached, not a similarity search over a snapshot that was true when the index was built.

The honest test is whether the question is answered by a passage of text that exists somewhere in a body of documents too large to send. If it is, retrieval fits. If the answer must be computed, or is about the user rather than about the documents, reach for a tool call instead, and let the model choose between them rather than forcing every question through a retriever.

Each stage above has a section of its own elsewhere in this track, and they are where the depth lives: parsing and splitting under chunking, exact and approximate search under vector databases, and rank fusion and cross-encoders under reranking. This section is deliberately the shape of the whole thing rather than the detail of any one part, because the commonest failure in production is not a weak stage, it is not knowing which stage is weak.

Where people go wrong

Treating retrieval as a fixed first step. Not every question needs retrieval, and some need several searches. Giving the model retrieval as a tool it can choose to call, possibly more than once, usually beats always retrieving exactly once.

Build this: Build RAG over twenty real documents with structural chunking and citations. Then deliberately ask a question the documents do not answer, and check whether the system says so or invents something.

Open this section on its own page

Section 2 of 2, 4h

Making it actually work

By the end: You can diagnose a failing RAG system and know which stage to fix.

Measuring retrieval separately from generation
Query rewriting
Handling the no-answer case
Keeping the index fresh

You can build one of these now. This section is about the harder part, which is fixing one that answers badly. There are only two places a bad answer can come from and they take completely different repairs, so the work starts by separating them and measuring each on its own. The rest covers rewriting queries before they are searched, letting the system say it does not know, and keeping an index from going stale.

Measure the stages separately#

A RAG system can fail at retrieval or at generation, and the fixes have nothing in common. Measure them apart or you will guess.

Retrieval:   was the correct chunk in the top k?
             a yes or no per question, over a fixed set

Generation:  given the correct chunk, was the answer right?
             feed the right chunk deliberately and score the answer

Two numbers. The first, recall@k, tells you whether search works, the second whether the answer stayed grounded in what it was given.

That is easier to agree with than to act on, so here it is on a corpus small enough to read: a twelve-section support handbook, thirteen questions with the chunk and the sentence that answers each one written down in advance, and a retriever and a generator that both run offline. The retriever is TF-IDF over character n-grams, which matches refund against refunds and stands in for an embedding model without needing one. The generator quotes the sentence in the context sharing most words with the question. Neither is as good as the real thing, and both fail in the directions the real things fail in, which is what the measurements are for.

import re
from sklearn.feature_extraction.text import TfidfVectorizer

HANDBOOK = """# Refunds
An approved refund is returned to the original payment method within 5 working days.
International refunds take up to 10 working days because of bank clearing.

# Returns
An item may be returned within 30 days of delivery if it is unused.
Sale items are returned under the same window as everything else.

# Delivery
Standard delivery arrives in 2 to 4 working days.
Express delivery is next day if the order is placed before 3pm.

# Addresses
We do not deliver to PO boxes.
A delivery address can be changed at any time before the order is dispatched.

# Accounts
The email address on an account is changed from the account settings page.
Deleting an account removes its order history after 30 days.

# Passwords
A password reset link is valid for 60 minutes.
Three failed sign-in attempts lock the account for 15 minutes.

# Payment
We accept cards and direct debit.
We do not accept cheques or bank transfers.

# Invoices
Invoices are issued on the first of each month.
A VAT receipt can be downloaded from the billing page.

# Damage
Damage in transit must be reported within 48 hours with a photograph.
A replacement is dispatched in the same week the report is accepted.

# Gift cards
A gift card is valid for 12 months from purchase.
Gift cards cannot be exchanged for cash.

# Subscriptions
A subscription can be cancelled at any time.
Cancellation takes effect at the end of the period already paid for.

# Contact
Support is open from 9am to 6pm on weekdays.
Live chat is on the help page.
"""

CHUNKS = []
for block in HANDBOOK.strip().split("\n\n"):
    lines = block.strip().split("\n")
    CHUNKS.append({"id": "c%d" % len(CHUNKS),
                   "heading": lines[0].lstrip("# ").strip(),
                   "text": " ".join(lines[1:]).strip()})
BY_ID = {c["id"]: c for c in CHUNKS}

# Character n-grams, so refund matches refunds. A stand-in for an embedding
# model: no network, same numbers on every run.
vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(4, 5), sublinear_tf=True)
INDEX = vec.fit_transform([c["heading"] + ". " + c["text"] for c in CHUNKS])

def retrieve(question, k=3):
    scores = (INDEX @ vec.transform([question]).T).toarray().ravel()
    return [(CHUNKS[i], float(scores[i])) for i in scores.argsort()[::-1][:k]]

STOP = set("a an and are be by can do does for from get have how i if in is it "
           "many me my of on or the to what when where will you your".split())

def keys(text):
    return {w[:4] for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}

def sentences(c):
    return [s.strip() + "." for s in c["text"].split(".") if s.strip()]

def generate(question, chunks):
    """Quote the sentence sharing most words with the question. Ties go to the
    first one seen, which is why the order of the context matters below."""
    want, best, high, cite = keys(question), None, 0, None
    for c in chunks:
        for s in sentences(c):
            n = len(want & keys(s))
            if n > high:
                best, high, cite = s, n, c["id"]
    return ("I do not know.", None) if best is None else (best, cite)

print(len(CHUNKS), "chunks,", sum(len(c["text"]) for c in CHUNKS), "characters")

# 12 chunks, 1352 characters

The whole corpus is 1,352 characters, which is the point: everything below can be checked by reading. Each section has two sentences on the same subject that differ in one detail, because that is where answering goes wrong.

# question, the chunk that should be found, which of its two sentences answers it
EVAL = [
    ("how long does a refund to my card take", "c0", 0),
    ("how long does a refund to an overseas bank take", "c0", 1),
    ("how long do I have to return an item", "c1", 0),
    ("can I return a discounted item", "c1", 1),
    ("when will standard delivery arrive", "c2", 0),
    ("how late can I order for next day delivery", "c2", 1),
    ("how long is a password reset link valid", "c5", 0),
    ("what happens after several failed logins", "c5", 1),
    ("how long is a gift card good for", "c9", 0),
    ("can I exchange a gift card for money", "c9", 1),
    ("if I cancel my subscription today when does it stop", "c10", 1),
    ("my parcel turned up smashed", "c8", 0),
    ("when can I speak to a human", "c11", 0),
]

def gold(chunk_id, i):
    return sentences(BY_ID[chunk_id])[i]

for k in (1, 3, 5):
    found = sum(g in [c["id"] for c, _ in retrieve(q, k)] for q, g, _ in EVAL)
    print("retrieval recall@%d          %.2f" % (k, found / len(EVAL)))

given = sum(generate(q, [BY_ID[g]])[0] == gold(g, i) for q, g, i in EVAL)
print("answers right given the gold chunk  %.2f" % (given / len(EVAL)))

end = sum(generate(q, [c for c, _ in retrieve(q, 3)])[0] == gold(g, i) for q, g, i in EVAL)
print("answers right end to end            %.2f" % (end / len(EVAL)))

for q, g, i in EVAL:
    got, _ = generate(q, [BY_ID[g]])
    if got != gold(g, i):
        print("  from the right chunk:", q)
        print("     said:", got)

# retrieval recall@1          0.85
# retrieval recall@3          0.92
# retrieval recall@5          0.92
# answers right given the gold chunk  0.62
# answers right end to end            0.62
#   from the right chunk: how long does a refund to my card take
#      said: International refunds take up to 10 working days because of bank clearing.
#   from the right chunk: can I return a discounted item
#      said: An item may be returned within 30 days of delivery if it is unused.
#   from the right chunk: if I cancel my subscription today when does it stop
#      said: A subscription can be cancelled at any time.
#   from the right chunk: my parcel turned up smashed
#      said: I do not know.
#   from the right chunk: when can I speak to a human
#      said: I do not know.

Retrieval finds the right chunk for 12 of 13 questions and the system answers 8 of them. Handed the correct chunk and nothing else, it still answers only 8. The gap is not a retrieval problem and no amount of index tuning closes it.

Read the three failures underneath. Asked how long a refund to a card takes, it quotes the sentence about international payments, because the word take appears in that sentence and not in the right one. Asked about a discounted item, it quotes the general returns rule rather than the sale-items sentence, because the document says sale and the question says discounted. Asked when a cancellation takes effect, it quotes the sentence saying a subscription can be cancelled at any time, which is true, adjacent and not an answer. Every one of these is the same shape: the right chunk, the wrong sentence in it. A single end-to-end score of 0.62 would have told you nothing about which of the two stages to work on, and the pair 0.92 and 0.62 tells you immediately.

Nothing relevant came back, and an answer arrived anyway#

The failure that costs the most trust is the one where retrieval returns nothing useful and the system answers from what the model already knew. Models do not experience an empty retrieval as missing information; they experience it as a question, and they are fluent. Here that behaviour is made explicit as a lookup table the stub falls back on when nothing in the context matches, so it can be switched off and measured rather than argued about.

# What the stub "learnt in training": plausible, unsourced, mostly wrong.
PRIOR = {
    "student": "Yes, students get 10% off with a valid card.",
    "app": "Yes, there is an app for iOS and Android.",
    "collect": "Yes, you can collect from any store.",
    "phone": "Support is on 0800 100 200.",
    "vat": "The VAT number is GB123456789.",
}
UNANSWERABLE = ["do you offer a student discount",
                "is there a mobile app",
                "can I collect my order from a shop",
                "what is your phone number",
                "what is your VAT number"]

def ask(question, k=3, floor=0.0, use_prior=True):
    hits = retrieve(question, k)
    if hits[0][1] < floor:
        return "I do not know."
    text, cite = generate(question, [c for c, _ in hits])
    if cite is None and use_prior:
        for trigger, fallback in PRIOR.items():
            if trigger in question.lower():
                return fallback
    return text

print("grounded only:")
for q in UNANSWERABLE:
    print("  %-34s %s" % (q, ask(q, use_prior=False)))
print("allowed to fall back on what it knows:")
for q in UNANSWERABLE:
    print("  %-34s %s" % (q, ask(q, use_prior=True)))

# grounded only:
#   do you offer a student discount    I do not know.
#   is there a mobile app              I do not know.
#   can I collect my order from a shop A delivery address can be changed at any
#                                      time before the order is dispatched.
#   what is your phone number          I do not know.
#   what is your VAT number            A VAT receipt can be downloaded from the
#                                      billing page.
# allowed to fall back on what it knows:
#   do you offer a student discount    Yes, students get 10% off with a valid card.
#   is there a mobile app              Yes, there is an app for iOS and Android.
#   can I collect my order from a shop A delivery address can be changed at any
#                                      time before the order is dispatched.
#   what is your phone number          Support is on 0800 100 200.
#   what is your VAT number            A VAT receipt can be downloaded from the
#                                      billing page.

Five questions the handbook cannot answer. With the fallback on, five answers come back and none of them is in the corpus. With it off, three become refusals and two do not, because the retriever found something adjacent: a question about collecting from a shop lands on the address-change rule, and a question about a VAT number lands on VAT receipts.

Those last two matter more than the invented ones. Forbidding the model to use outside knowledge does not stop a confident wrong answer, because the retriever handed it real text on a neighbouring subject and answering from that text is exactly what it was told to do. Grounding is not a defence against irrelevance. What catches these is the score, not the instruction.

A floor, and what the floor costs#

answerable = [retrieve(q, 1)[0][1] for q, _, _ in EVAL]
unanswerable = [retrieve(q, 1)[0][1] for q in UNANSWERABLE]
print("answerable    min %.2f  median %.2f  max %.2f"
      % (min(answerable), sorted(answerable)[len(answerable) // 2], max(answerable)))
print("unanswerable  min %.2f  median %.2f  max %.2f"
      % (min(unanswerable), sorted(unanswerable)[len(unanswerable) // 2], max(unanswerable)))
print("floor  unanswerable let through  answerable blocked")
for floor in (0.00, 0.15, 0.20, 0.30, 0.40):
    print("%.2f   %d of 5                   %d of 13"
          % (floor, sum(s >= floor for s in unanswerable), sum(s < floor for s in answerable)))

# answerable    min 0.15  median 0.45  max 0.66
# unanswerable  min 0.08  median 0.13  max 0.28
# floor  unanswerable let through  answerable blocked
# 0.00   5 of 5                   0 of 13
# 0.15   2 of 5                   0 of 13
# 0.20   1 of 5                   2 of 13
# 0.30   0 of 5                   3 of 13
# 0.40   0 of 5                   5 of 13

The two distributions overlap, so no floor separates them cleanly. A floor at 0.15 removes three of the five bad answers and blocks nothing good. Removing the fourth costs two good questions and removing the fifth costs three, and raising it further only blocks more. This curve is cheap to produce and is the only honest way to pick the number.

So the threshold is a product decision with a measured price, not a constant to copy from a tutorial. For a support bot, refusing a question the handbook covers is a smaller harm than inventing a VAT number, and the floor goes high. For a research assistant used by people who check everything, a wrong answer costs a minute and a refusal costs a session, and the floor goes low. What is not defensible is choosing 0.7 because it looked confident, on a score whose scale you never plotted.

Right, and unsupported#

Correctness and groundedness are different properties and only one of them is under your control. An answer can be true and absent from the retrieved text, which means the system got lucky rather than working, and the same code will invent the next one. Measuring only accuracy hides that completely.

def supported(text, chunks):
    """Is every content word of the answer present in the retrieved text?"""
    context = set()
    for c in chunks:
        context |= keys(c["text"])
    return keys(text) <= context

# True where the stub's answer happens to be factually right anyway.
TRUTH = {"do you offer a student discount": False,
         "is there a mobile app": False,
         "can I collect my order from a shop": False,
         "what is your phone number": True,
         "what is your VAT number": False}

grid = {}
for q in UNANSWERABLE:
    out = ask(q, use_prior=True)
    cell = (TRUTH[q], supported(out, [c for c, _ in retrieve(q, 3)]))
    grid[cell] = grid.get(cell, 0) + 1
    print("right=%-5s supported=%-5s  %s" % (cell[0], cell[1], out))
print()
for right in (True, False):
    for sup in (True, False):
        print("right=%-5s supported=%-5s  %d" % (right, sup, grid.get((right, sup), 0)))

# right=False supported=False  Yes, students get 10% off with a valid card.
# right=False supported=False  Yes, there is an app for iOS and Android.
# right=False supported=True   A delivery address can be changed at any time
#                              before the order is dispatched.
# right=True  supported=False  Support is on 0800 100 200.
# right=False supported=True   A VAT receipt can be downloaded from the billing page.
#
# right=True  supported=True   0
# right=True  supported=False  1
# right=False supported=True   2
# right=False supported=False  2

Word containment is a crude support check and it is enough to separate the cells. The phone number is the interesting row: factually right, nowhere in the sources, and indistinguishable from the invented ones by any test that only asks whether the answer was correct.

In a real evaluation the support check is usually another model asked whether each claim follows from the retrieved text, which is a judge and needs validating like any other judge. The shape of the measurement does not change: one score for whether the answer is right, a second for whether the retrieved text supports it, and a rule that an unsupported answer is a defect even when it is correct. Teams that report a single quality number ship the phone-number row and find out from a customer.

The order the context is assembled in#

Retrieval returns chunks with scores and the prompt is a flat string, so something has to decide what goes in and in what order. That decision is usually made by accident, in a line that sorts the chunks for readability and then trims whatever does not fit the budget. Those two steps in that order throw away the best evidence.

def fit(chunks, budget=400):
    """Keep chunks in the order given, until the budget runs out."""
    kept, used = [], 0
    for c in chunks:
        if used + len(c["text"]) <= budget:
            kept.append(c)
            used += len(c["text"])
    return kept

by_score = lambda hits: [c for c, _ in hits]
in_document_order = lambda chunks: sorted(chunks, key=lambda c: int(c["id"][1:]))

def run(select, present, label):
    kept_gold = right = 0
    for q, g, i in EVAL:
        context = present(fit(select(retrieve(q, 6))))
        kept_gold += g in [c["id"] for c in context]
        right += generate(q, context)[0] == gold(g, i)
    print("%-46s gold kept %2d/13  right %2d/13" % (label, kept_gold, right))

run(by_score, lambda cs: cs, "trim in score order, best chunk first")
run(lambda h: in_document_order(by_score(h)), lambda cs: cs, "trim in document order")
run(by_score, lambda cs: cs[::-1], "trim in score order, best chunk last")
run(by_score, in_document_order, "trim in score order, present in document order")

# trim in score order, best chunk first          gold kept 12/13  right  8/13
# trim in document order                         gold kept  9/13  right  6/13
# trim in score order, best chunk last           gold kept 12/13  right  7/13
# trim in score order, present in document order gold kept 12/13  right  7/13

Sorting into document order before trimming drops the best chunk for 3 of 13 questions and costs two answers. Selecting by score first and reordering afterwards keeps all 12 whatever order they are then printed in. Selection and presentation are two decisions and only the first one can lose information.

The rule that falls out is short. Choose what fits by score, then choose the order for the reader's sake, and never let a sort run between the two. The remaining difference in that table, between 8 and 7 answers, comes from where the best chunk sits once everything has been selected, and the stub is a poor model of that: it breaks ties by position, which is a crude version of a real effect. Models are measurably better at using material at the beginning and the end of a long context than material buried in the middle, an effect documented well enough to plan around. With a handful of chunks it does not matter. With forty it does, and the cheap defence is to send fewer chunks rather than to guess at an ordering.

Query rewriting#

Users do not write good search queries. They write 'what about the other thing' on turn four. Rewriting the query using the conversation before searching often helps more than any tuning of the index.

History:  "How long do refunds take?"
          "Five working days."
User:     "and for international?"

Rewritten: "How long do international refunds take?"

The no-answer case#

The failure that destroys trust

A RAG system confidently answering a question its documents do not cover is worse than one that says it does not know. Users discover the invention once and stop trusting every answer after it.

Set a relevance threshold. If the best chunk scores below it, do not answer from retrieval at all.
Instruct the model explicitly that declining is a correct answer when the sources do not cover the question.
Put no-answer cases in your eval set. They are the ones nobody tests and the ones that damage trust.

Freshness#

Documents change. An index built once and never updated slowly becomes a confident source of outdated answers, which is a particularly hard failure to notice because nothing errors.

Re-index changed documents rather than rebuilding everything, keyed on a content hash.
Store a last-updated date on each chunk and show it with the citation.
Delete chunks for documents that were removed, or they will keep being cited.

Where people go wrong

Evaluating the whole pipeline with one number. It tells you something is wrong and nothing about where, so every fix is a guess.

Build this: Build separate retrieval and generation evals over thirty questions, including five the documents cannot answer. Report recall at 5, generation accuracy given correct context, and how often the system wrongly answers a no-answer question.

Open this section on its own page

Where you use this

A topic is easier to care about when you can see what it is for. These projects lean on it at a named stage.

RAG with citations you can check

Stage 3, Read what comes back: Which stage you are looking at, and what each later stage can and cannot repair.
Stage 7, Say nothing rather than something: What the abstain path is for, and where it sits relative to everything else that helps.

Now actually do it

Reading a topic is not learning it. These are the hours that count, and each one tells you when you are finished.

1Ingest something genuinely messy

Twenty real documents including at least one multi-column PDF, one with tables and one scanned. Extract, inspect the output by eye, and write a check that flags documents that extracted badly.

Done when: You can name which documents your pipeline should refuse to index, and it refuses them.

2Chunk three ways and measure

Fixed size, structural, and structural with heading-path enrichment before embedding. Run the same 20-question retrieval eval on each.

Done when: You have recall@5 for all three and can say which won and by how much.

3Hybrid search and reranking, measured

Add keyword search fused by rank, then a reranker. Measure recall@5 and latency after each step.

Done when: A table showing recall and latency for vector only, hybrid, and hybrid plus reranking.

4Handle the no-answer case

Add five questions your documents cannot answer. Add a relevance threshold and an instruction that declining is correct. Measure how often the system wrongly answers them.

Done when: The system declines on all five, and you have the number it declined at before tuning.

14 hours of practice

Words used here

3 terms this topic teaches, each with a one-line definition if one of them stopped you.

Retrieval-augmented generation: Fetching relevant material at question time and putting it in the prompt, so the model answers from supplied text rather than from memory. Knowledge can then be updated, restricted and cited without retraining.
Recall@k: The fraction of questions whose answer appears somewhere in the top k retrieved passages. The number that tells you whether a retrieval problem is worth working on at all.
Grounding: Tying a model's answer to specific supplied text, so every claim can be traced to a passage that supports it.

Interview questions

8 questions, grouped by the section that teaches the answer. An answer you have read is not an answer you can give, so use Test yourself and say it out loud first.

The RAG pipeline

What problem does RAG solve?

Models only know their training data, so they cannot answer about your private or recent documents, and they tend to invent an answer rather than decline. RAG retrieves the relevant text and puts it in the prompt so the answer comes from evidence.

How would you chunk documents?

On structure rather than a fixed character count: paragraphs and sections, keeping headings attached to their content, with slight overlap between chunks. Store the document title and heading path with each chunk and include them when embedding, so a chunk is self-describing.

Why ask the model to cite sources?

It lets users verify, it lets you debug which chunk caused a wrong answer, and it reduces invention because the model is less willing to assert something it must attribute.

When would you fine-tune instead of using RAG?

Fine-tuning teaches form: a consistent output format, a domain vocabulary, a house style. RAG supplies facts. If the problem is that the model does not know something, RAG is the answer, because fine-tuning does not reliably install new facts and cannot be updated without retraining.

Then they push further

A team wants to fine-tune a model on their docs so it knows the product. Good idea?

Usually not. Fine-tuning is unreliable at adding facts, cannot be updated when a document changes without training again, and cannot cite a source. Retrieval keeps knowledge current and checkable. Fine-tuning fits problems of behaviour, format or style, not knowledge.

Can you use both?

Yes. Retrieve for the facts and fine-tune for how to use them: a house citation style, a strict output format, or making a small model good enough at the task to replace a larger one.

What gets marked up. Strong answers separate knowledge from behaviour and mention keeping content current and citing sources. Weak answers treat fine-tuning as the more serious or advanced option.

Making it actually work

Your RAG system gives a wrong answer. How do you find out why?

Separate the stages. Check whether the correct chunk was retrieved. If it was not, the problem is chunking, embedding or ranking. If it was, feed it deliberately and check the answer, which isolates a prompt or model problem.

How do you handle a question your documents cannot answer?

Threshold on retrieval score so a weak best match does not proceed, instruct the model that declining is acceptable, and include no-answer cases in the eval set. A confident wrong answer costs more trust than a refusal.

Then they push further

How do you measure how well it declines?

Put unanswerable questions in the eval set on purpose and measure how often it answers them anyway, alongside how often it wrongly declines questions it could answer. Improving one usually worsens the other, so track both.

What gets marked up. Marked up for treating 'I do not know' as a correct answer with its own test cases, and for tracking wrong refusals too.

What is query rewriting and why does it matter?

Rewriting the user's question into a standalone search query using the conversation context. Follow-up questions like 'and for international?' are meaningless to a search index on their own, and rewriting often improves retrieval more than index tuning.

How do you keep a RAG index current?

Re-index on change rather than rebuilding, keyed by content hash. Delete chunks for removed documents. Store and display a last-updated date, because a stale answer is indistinguishable from a correct one to the user.

These questions on their own page, one link per question