The RAG pipeline

Retrieval-augmented generation is the standard way to make a model answer from your material rather than from whatever it absorbed in training. Find the relevant passages, put them in the prompt, ask the question there. This section is the whole thing end to end, from splitting documents through to citing where an answer came from, and it gives most of its attention to chunking, which is dull and decides more of the final quality than the choice of model does.

The idea#

A model knows what was in its training data. It does not know your documents, and it will not admit that gracefully. RAG fixes this by finding the relevant text and putting it in the prompt, so the model answers from evidence in front of it rather than from memory.

Question  ->  retrieve relevant chunks  ->  put them in the prompt
          ->  model answers using them  ->  answer plus citations

The stages#

1. 1.Ingest: parse documents into clean text. PDFs, HTML and tables all fight you here.
2. 2.Chunk: split into pieces small enough to retrieve precisely and large enough to make sense alone.
3. 3.Embed and index: convert chunks to vectors and store them alongside the original text and metadata.
4. 4.Retrieve: hybrid search, then rerank, to get a handful of good chunks.
5. 5.Generate: put those chunks in the prompt and ask the model to answer using only them.

The whole thing, running#

Each of those stages is a function you can write yourself, and the version that fits on one screen is worth building before adopting anything that hides it. What follows is a complete pipeline: four documents, a chunker, an index, retrieval, prompt assembly and a generator, none of which touches a network. The generator is a stub that quotes whichever sentence in the retrieved text shares most words with the question, so the pipeline runs offline and gives the same answer every time. Replacing it with a model call changes one function and leaves the other four alone, which is the useful thing to see.

import re
from sklearn.feature_extraction.text import TfidfVectorizer

HANDBOOK = """# Refunds
An approved refund is returned to the original payment method within 5 working days.
International refunds take up to 10 working days because of bank clearing.

# Returns
An item may be returned within 30 days of delivery if it is unused.
Sale items are returned under the same window as everything else.

# Delivery
Standard delivery arrives in 2 to 4 working days.
Express delivery is next day if the order is placed before 3pm.

# Accounts
The email address on an account is changed from the account settings page.
Deleting an account removes its order history after 30 days.
"""

def chunk(document):
    """One chunk per heading, with the heading kept on the text."""
    out = []
    for block in document.strip().split("\n\n"):
        lines = block.strip().split("\n")
        out.append({"id": "c%d" % len(out),
                    "heading": lines[0].lstrip("# ").strip(),
                    "text": " ".join(lines[1:]).strip()})
    return out

CHUNKS = chunk(HANDBOOK)
for c in CHUNKS:
    print(c["id"], c["heading"], len(c["text"]), "chars")

vec = TfidfVectorizer(stop_words="english")
INDEX = vec.fit_transform([c["heading"] + ". " + c["text"] for c in CHUNKS])
print("index:", INDEX.shape[0], "chunks by", INDEX.shape[1], "terms")

# c0 Refunds 159 chars
# c1 Returns 133 chars
# c2 Delivery 113 chars
# c3 Accounts 135 chars
# index: 4 chunks by 38 terms

Retrieval is a similarity and a sort. Assembly is string formatting. Neither is clever, and both are where systems go wrong, so it is worth watching what they actually produce rather than trusting a library to produce it.

def retrieve(question, k=2):
    scores = (INDEX @ vec.transform([question]).T).toarray().ravel()
    return [(CHUNKS[i], float(scores[i])) for i in scores.argsort()[::-1][:k]]

for c, s in retrieve("how long does a refund take", k=3):
    print("%.3f  %s  %s" % (s, c["id"], c["heading"]))

# 0.241  c0  Refunds
# 0.000  c3  Accounts     <- asked for three, so three came back
# 0.000  c2  Delivery

def assemble(question, hits):
    lines = ["Answer using only the sources below.",
             "Cite the source id in brackets after each claim.",
             "If the sources do not answer the question, say you do not know.", ""]
    lines += ["[%s] %s: %s" % (c["id"], c["heading"], c["text"]) for c, _ in hits]
    lines += ["", "Question: " + question]
    return "\n".join(lines)

print(assemble("how long does a refund take", retrieve("how long does a refund take")))

# Answer using only the sources below.
# Cite the source id in brackets after each claim.
# If the sources do not answer the question, say you do not know.
#
# [c0] Refunds: An approved refund is returned to the original payment method
#      within 5 working days. International refunds take up to 10 working days
#      because of bank clearing.
# [c3] Accounts: The email address on an account is changed from the account
#      settings page. Deleting an account removes its order history after 30 days.
#
# Question: how long does a refund take

The last stage is the model, and a stub with known behaviour teaches more here than an API key would. This one extracts rather than composes, which is cruder than a real model but fails in a recognisable direction, and it declines when retrieval came back with nothing above a floor.

STOP = set("a an and are be by can do does for from get have how i if in is it "
           "many me my of on or the to what when where will you your".split())

def keys(text):
    """Four-letter prefixes, so refund and refunds match."""
    return {w[:4] for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}

def generate(question, hits):
    """Stands in for the model: quote the sentence sharing most words with the
    question, and cite the chunk it came from. Deterministic, offline."""
    want, best, high, cite = keys(question), None, 0, None
    for c, _ in hits:
        for sentence in [s.strip() + "." for s in c["text"].split(".") if s.strip()]:
            overlap = len(want & keys(sentence))
            if overlap > high:
                best, high, cite = sentence, overlap, c["id"]
    if best is None:
        return "I do not know."
    return "%s [%s]" % (best, cite)

def ask(question, k=2, floor=0.15):
    hits = retrieve(question, k)
    if hits[0][1] < floor:
        return "I do not know. Best chunk scored %.3f, under the floor." % hits[0][1]
    return generate(question, hits)

for q in ["how many days do I have to return an item",
          "how long do international refunds take",
          "how long does a refund take",
          "do you ship to Ireland"]:
    print(q)
    print("   ", ask(q))

# how many days do I have to return an item
#     An item may be returned within 30 days of delivery if it is unused. [c1]
# how long do international refunds take
#     International refunds take up to 10 working days because of bank clearing. [c0]
# how long does a refund take
#     International refunds take up to 10 working days because of bank clearing. [c0]
# do you ship to Ireland
#     I do not know. Best chunk scored 0.000, under the floor.

That third answer is worth staring at, because it is the failure this topic is mostly about. Retrieval did its job perfectly. The correct sentence was in the prompt. The answer is still wrong, and no change to the index, the embedding model or the chunk size would have prevented it. A system measured by a single end-to-end score would record that as a retrieval problem and send you to tune the wrong half. The next section splits the measurement so it cannot.

Chunking is where quality is won or lost#

Splitting every 500 characters is the default advice and it is bad. It cuts sentences in half, separates a heading from the text under it, and splits tables down the middle. The chunk then reads as nonsense and cannot be retrieved sensibly.

Bad, split at a fixed character count:

  chunk 41: "...and the refund period is"
  chunk 42: "5 working days from receipt..."

Neither chunk answers "how long do refunds take".

Split on structure instead. Paragraphs, sections, headings. Keep the heading with its content, and attach the document title to every chunk so a fragment still carries its context.

DOC = {"title": "Support handbook", "url": "https://example.com/handbook"}

enriched = [{"text": c["text"],
             "title": DOC["title"],            # so the chunk is self-describing
             "heading": c["heading"],
             "source": DOC["url"],
             "position": position}
            for position, c in enumerate(CHUNKS)]

# What gets embedded includes the context, not just the raw text.
def embedding_text(chunk):
    return "%s > %s\n\n%s" % (chunk["title"], chunk["heading"], chunk["text"])

print(embedding_text(enriched[0]))

# Support handbook > Refunds
#
# An approved refund is returned to the original payment method within 5 working
# days. International refunds take up to 10 working days because of bank clearing.

Citations#

Give each chunk an identifier and ask the model to cite the ones it used. This gives the user a way to verify, gives you a way to debug, and measurably reduces invention, because a model asked to cite is less inclined to assert something it cannot point at.

Answer using only the sources below. After each claim, cite the
source id in square brackets. If the sources do not contain the
answer, say so and cite nothing.

[doc_12] Refunds are processed within 5 working days.
[doc_37] Returns are accepted within 30 days of delivery.

Citations also change what you can debug. An answer with a source id can be checked against that source mechanically, which is the difference between knowing an answer is wrong and knowing which stage made it wrong. An answer without one can only be argued about.

When RAG is the wrong tool#

Retrieval is the default answer to so many questions that it gets applied to problems it does not fit, and the resulting system is slower, more expensive and worse than the thing it replaced. Four cases come up repeatedly.

• The corpus is small enough to send. Twenty pages of policy is a few thousand tokens. Putting all of it in the prompt has perfect recall, no index to maintain and no retrieval stage to debug, and prompt caching makes the repeated cost small. Retrieval earns its complexity when the corpus cannot fit, not when it is merely large-ish.
• The question is not a lookup. Aggregates, comparisons and counts are database queries wearing a question mark. How many orders were refunded last month is answered by SQL, and a retriever that returns the five most similar order records will answer it confidently and wrongly. Give the model the query tool instead.
• The problem is behaviour, not knowledge. A consistent output format, a house tone, a classification the model keeps getting wrong: none of these are fixed by putting documents in the prompt. Prompting, examples or fine-tuning address those, and retrieval adds latency without touching the cause.
• The answer changes per user and lives in a system of record. Account balance, order status, entitlement. These want an API call with the user's identity attached, not a similarity search over a snapshot that was true when the index was built.

The honest test is whether the question is answered by a passage of text that exists somewhere in a body of documents too large to send. If it is, retrieval fits. If the answer must be computed, or is about the user rather than about the documents, reach for a tool call instead, and let the model choose between them rather than forcing every question through a retriever.

Each stage above has a section of its own elsewhere in this track, and they are where the depth lives: parsing and splitting under chunking, exact and approximate search under vector databases, and rank fusion and cross-encoders under reranking. This section is deliberately the shape of the whole thing rather than the detail of any one part, because the commonest failure in production is not a weak stage, it is not knowing which stage is weak.