97 - RAG From Scratch: Retrieving the Right Context, and Answers With a Verifiable Citation

A language model knows what it read during training, and knows nothing about your documents. Retrieval-augmented generation (RAG) solves the problem in the most direct way: before asking the model to answer, you look for the pieces of document that contain the answer and paste them into the prompt. The idea has a name and a date - Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401 - and it has become the most widely used architecture for making a model talk about things it has never seen.

It is also the architecture where the diagnosis is most often wrong: when a RAG system answers badly, instinct sends you off to rewrite the prompt, and often the prompt is not the problem. Here we build a RAG piece by piece on a real corpus, we measure every decision - how to cut the documents, how many pieces to pass, whether to rerank them, whether to filter them by metadata - and we add the thing that is almost always missing: a check that for every claim goes back to the cited span and refuses the answer if numbers and names do not appear literally in there.

How the examples are written, and what they can demonstrate. No model runs in this lesson. Retrieval uses a fake encoder: a deterministic hashing of words weighted with the IDF, which has learned nothing and therefore does not understand synonyms. Generation uses a scripted generator: a function that takes the sentence of the spans with the most words in common with the question and cites it. Neither is a model, and I say so because the figures that come out measure the method (where a RAG breaks, how you see it, how you repair it), not the quality of a real model: with a trained encoder the numbers would change, the shape of the problems would not. The advantage is that everything is rerunnable and identical on every run. Everything runs on Python 3.14.4 and NumPy 2.3.5; the corpus comes from the docstrings of the standard library of that version, so with another version the numbers will be close but not equal.

The analogy: the open-book exam

Picture a well-prepared student at an open-book exam. They have understood the subject, they reason well, they can write. The book is on the desk. The student opens it at three pages, reads them, and answers using only those.

If the three pages are the right ones the answer is excellent: there is nothing to remember, only reading and summarising. If they are the wrong ones - a neighbouring topic, an adjacent chapter, but not the page with the definition asked for - one of two things happens, both failures: either they admit that those pages do not answer, or they build a plausible answer out of what they have in front of them. No improvement in their reasoning saves them, because the limit is in the choice of pages.

In a RAG the model is the student and retrieval is the hand that opens the book. Hence the thesis of this lesson, which is worth writing in one line: the generator cannot repair a wrong retrieval. It is not an opinion about architecture, it is a measurable limit, and the first skill to learn is to measure it in a minute instead of spending an afternoon on the prompt.

The three phases, and the number to look at before any other

A RAG has three phases. Indexing, once only: the documents are cut into pieces (chunks), each piece becomes a vector, the vectors go into an index. Retrieval, at every question: the question becomes a vector and the index returns the k nearest pieces. Generation: the k pieces are laid out in a prompt and the model answers.

The typical failure sits in the second phase and shows up in the third: that is why it is diagnosed wrongly. The measure that separates them is called the ceiling: the percentage of questions for which the answer is inside the k chunks handed to the model. If the ceiling is 0.50, half of the questions reach the generator without the answer in the context, and on that half the model can only abstain or invent. It is a hard upper limit.

The corpus, the golden set and the ceiling in ten lines

To measure the ceiling you need documents and questions with a known answer. The corpus is built offline: each document is a module of the standard library, that is its docstring plus those of the public functions and classes it defines, concatenated. They are long documents like real ones, not fifty-word fragments, and it is their length that makes segmentation a decision.

import collections, hashlib, importlib, inspect, math, re, statistics, time
import numpy as np

MODULES = ["statistics", "bisect", "heapq", "itertools", "functools", "difflib",
           "textwrap", "tempfile", "uuid", "base64", "secrets", "gzip", "math",
           "glob", "zlib", "operator", "fractions", "fnmatch"]

def normalize(s):
    return " ".join(s.split())

def build_corpus():
    documents, seen = [], set()
    for name in MODULES:
        m = importlib.import_module(name)
        parts = [f"MODULE {name}"]
        if inspect.getdoc(m):
            parts.append(inspect.getdoc(m).strip())
        for attr, obj in sorted(vars(m).items()):
            if attr.startswith("_"):
                continue
            if not (inspect.isfunction(obj) or inspect.isclass(obj) or inspect.isbuiltin(obj)):
                continue
            doc = inspect.getdoc(obj)
            if not doc or len(doc.split()) < 20:
                continue
            home = obj.__module__.lstrip("_")               # the module that defines it
            if home != name and home in MODULES:            # re-exported from a module that
                continue                                    # is in the corpus: its owner keeps it
            fingerprint = (obj.__module__, obj.__qualname__)   # and only once
            if fingerprint in seen:
                continue
            seen.add(fingerprint)
            parts.append(f"## {name}.{attr}\n\n{doc.strip()}")
        documents.append({"id": name, "text": "\n\n".join(parts), "status": "current"})
    return documents

Two details of the code prevent silent errors. The check on __module__ excludes the functions that a module imports from another: statistics imports bisect_left and Fraction, and without the check the first module in the list would keep them, leaving bisect and fractions without their own functions (it is a defect I found exactly that way, with an anchor that could not be found). And the twenty-word filter discards one-line docstrings, which do not contain an answer but only a label.

Then the golden set. Each question carries an anchor: the literal string, taken from the docstring, that contains the answer. A chunk counts as correct only if it contains all of it. It is strict and for that reason useful: it distinguishes "the chunk comes from the right document" from "the chunk contains the sentence that answers", two things that get confused all the time. To the eighteen questions I added ten hand-written notes that state the opposite of the library: drafts and outdated versions, marked in the status field. A corpus without contradictions is a fake corpus: in real repositories the current documentation lives alongside the documentation of three versions ago and someone's notes.

# (question, module, anchor): the anchor is the literal string that contains the answer
GOLDEN = [
    ("What is the default cutoff of get_close_matches?", "difflib",
     "Optional arg cutoff (default 0.6)"),
    ("Which position does bisect_left return when x is already in the list?", "bisect",
     "insert just before the leftmost x already there"),
    ("Where does insort_right put an item that is already present?", "bisect",
     "insert it to the right of the rightmost x"),
    ("How does heappushpop compare with heappush followed by heappop?", "heapq",
     "runs more efficiently than heappush() followed by a separate call to heappop()"),
    ("What does fmean do when the dataset is empty?", "statistics",
     "If the input dataset is empty, it raises a StatisticsError."),
    ("What is the average speed from harmonic_mean for 40 km/hr and 60 km/hr over equal distances?", "statistics",
     "harmonic_mean([40, 60]) 48.0"),
    ("What does median_low return with an even number of data points?", "statistics",
     "the smaller of the two middle values is returned"),
    ("How does textwrap shorten treat whitespace in the text?", "textwrap",
     "The text first has its whitespace collapsed."),
    ("Who can read the directory created by mkdtemp?", "tempfile",
     "readable, writable, and searchable only by the creating user"),
    ("Is a NamedTemporaryFile deleted automatically by default?", "tempfile",
     "whether the file is automatically deleted (default True)"),
    ("Why use compare_digest instead of comparing secrets with ==?", "secrets",
     "designed to prevent timing analysis"),
    ("What happens to the last batch of itertools batched when the input runs out?", "itertools",
     "The last batch may be shorter than n."),
    ("What does lru_cache do when maxsize is None?", "functools",
     "the LRU features are disabled and the cache can grow without bound"),
    ("Which characters replace plus and slash in urlsafe_b64encode?", "base64",
     "uses '-' instead of '+' and '_' instead of '/'"),
    ("Does the star wildcard of glob match filenames that start with a dot?", "glob",
     "filenames starting with a dot are special cases that are not matched by '*' and '?'"),
    ("Does gzip compress store the modification time of the data?", "gzip",
     "The modification time is set to 0 by default, for reproducibility."),
    ("What does math comb return when k is greater than n?", "math",
     "evaluates to zero when k > n"),
    ("What rational number does Fraction(8, 6) produce?", "fractions",
     "Fraction(8, 6) will produce a rational number equivalent to 4/3"),
]
QUESTIONS = [g[0] for g in GOLDEN]
ANCHORS = [g[2] for g in GOLDEN]

# ten hand-written notes that state the opposite: drafts or outdated versions
NOTES = [
    ("note-difflib", "draft", "NOTE (draft): the default cutoff of get_close_matches is 0.9, so only near identical words are returned."),
    ("note-bisect-1", "outdated", "NOTE (old version): bisect_left returns the position after the rightmost x already in the list when x is already in the list."),
    ("note-bisect-2", "draft", "NOTE (draft): insort_right puts an item that is already present before the leftmost equal item."),
    ("note-heapq", "outdated", "NOTE (old version): heappushpop is slower than heappush followed by heappop, and the two differ in cost."),
    ("note-statistics-1", "draft", "NOTE (draft): fmean returns 0.0 when the dataset is empty and raises nothing."),
    ("note-statistics-2", "draft", "NOTE (draft): median_low with an even number of data points returns the larger of the two middle values."),
    ("note-tempfile-1", "outdated", "NOTE (old version): the directory created by mkdtemp is readable by every user of the machine."),
    ("note-tempfile-2", "draft", "NOTE (draft): a NamedTemporaryFile is never deleted automatically by default, call os.remove yourself."),
    ("note-functools", "outdated", "NOTE (old version): lru_cache with maxsize None keeps the LRU features enabled and evicts the oldest entry."),
    ("note-gzip", "draft", "NOTE (draft): gzip compress stores the current time as the modification time of the data."),
]

The rest of the construction builds the corpus, checks every anchor and adds the notes; the block then prints the sizes that matter for reading the later measurements (a corpus with documents of 230 words behaves differently from one with documents of 5000 words).

documents = build_corpus()
by_id = {d["id"]: d for d in documents}
for question, module, anchor in GOLDEN:
    assert normalize(anchor) in normalize(by_id[module]["text"]), (module, anchor)
documents += [{"id": i, "text": t, "status": s} for i, s, t in NOTES]

doc_words = [len(d["text"].split()) for d in documents]
print(f"documents: {len(documents) - len(NOTES)} modules + {len(NOTES)} notes = {len(documents)}")
print(f"total words {sum(doc_words)} | median per document {int(statistics.median(doc_words))} | max {max(doc_words)}")
print(f"questions in the golden set: {len(GOLDEN)}")

documents: 18 modules + 10 notes = 28
total words 12822 | median per document 230 | max 2989
questions in the golden set: 18

The golden set has 18 questions, and that is few: I say so instead of hiding it. The minimum step is 1/18 ≈ 0.056, so a percentage point does not exist, and a difference of one question in eighteen does not separate two configurations. The assert at the end of the construction is the part not to skip: if an anchor does not appear in the document, the question is invalid, and the ceiling would be underestimated because of your mistake and not retrieval's. The serious sizing of a golden set is the subject of the lesson on evaluating generative systems; here it serves to show that you measure, and how.

The fake encoder, the index and the ceiling

The fake encoder does three things: it splits the text into words (separating camelCase too), removes the stopwords, and projects each word into one of the 1024 positions of the vector with a sign chosen by the hash. The weight is (1 + log c) · idf, where the IDF is computed on the chunks of the index: that is the reason why fit has to be called with the chunks of that configuration. It is not an approximation of a real model: it is lexical retrieval dressed up as a vector, and should be read that way. A trained encoder would also find "star" for "asterisk"; this one does not.

STOP = set("a an and are as at be by for from if in into is it its of on or that the this to when with "
           "what which who how does do not can than then there their was were will".split())

def words(text):
    text = re.sub(r"([a-z])([A-Z])", r"\1 \2", text)             # camelCase -> camel Case
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]

def _h(s):
    return int.from_bytes(hashlib.blake2b(s.encode(), digest_size=8).digest(), "big")

class FakeEncoder:
    """FAKE encoder: signed hashing of words, weighted with the IDF of the chunks. No model."""
    def __init__(self, dim=1024):
        self.dim, self.idf, self.idf_unknown = dim, {}, 1.0

    def fit(self, texts):
        df = collections.Counter()
        for t in texts:
            df.update(set(words(t)))
        n = len(texts)
        self.idf = {w: math.log((1 + n) / (1 + c)) + 1 for w, c in df.items()}
        self.idf_unknown = math.log(1 + n) + 1      # a word never seen in the chunks: the rarest
        return self

    def encode(self, texts):
        E = np.zeros((len(texts), self.dim), dtype=np.float32)
        for r, t in enumerate(texts):
            for w, c in collections.Counter(words(t)).items():
                h = _h(w)
                E[r, h % self.dim] += (1 if (h >> 40) & 1 else -1) * (1 + math.log(c)) * self.idf.get(w, self.idf_unknown)
        n = np.linalg.norm(E, axis=1, keepdims=True)
        return E / np.where(n == 0, 1, n)

Before the search you need the chunkers, that is the three strategies for cutting a document. Then two functions: the index and the search. The delicate point is the filter: search applies it before sorting, setting the score of the excluded ones to minus infinity, so that they do not compete for the k positions. Filtering afterwards - retrieving the top k and then discarding - would leave fewer than k useful results, or none.

def fixed_window(text, size, overlap):
    """Windows of `size` words that slide by `size - overlap`."""
    words_ = text.split()
    step = max(1, size - overlap)                   # max(1, ...): with overlap >= size the
    return [" ".join(words_[i:i + size])            # step would be zero and range() would stop
            for i in range(0, len(words_), step)    # with an error instead of advancing
            if words_[i:i + size]] or [text]

def by_structure(text, max_size):
    """Cut on paragraph boundaries; merge short paragraphs up to max_size words."""
    paragraphs = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
    pieces, current = [], []
    for p in paragraphs:
        if current and sum(len(x.split()) for x in current) + len(p.split()) > max_size:
            pieces.append("\n\n".join(current))
            current = []
        current.append(p)
    return pieces + (["\n\n".join(current)] if current else [])

def by_sentence(text, sentences_per_piece, overlapping_sentences):
    """A window of N sentences that slides by N minus the overlapping sentences."""
    sentences = [f.strip() for f in re.split(r"(?<=[.!?:])\s+", text.replace("\n", " ")) if f.strip()]
    step = max(1, sentences_per_piece - overlapping_sentences)
    return [" ".join(sentences[i:i + sentences_per_piece])
            for i in range(0, len(sentences), step) if sentences[i:i + sentences_per_piece]] or [text]

The fixed window is the simplest and the blindest: it cuts where the count falls. Segmentation by structure respects the paragraphs and merges the short ones up to the word cap; it is the most sensible if the text has paragraphs, and docstrings do. The one by sentence slides the window by sentences rather than words, and produces small, numerous pieces. The max(1, ...) in the steps is not pedantry: with an overlap equal to the window the step would be zero and range would stop with an error, in the middle of an indexing run.

def index_docs(documents, chunker):
    chunks = [{"text": t, "doc": d["id"], "status": d["status"]}
              for d in documents for t in chunker(d["text"])]
    enc = FakeEncoder().fit([c["text"] for c in chunks])
    return {"chunks": chunks, "enc": enc, "M": enc.encode([c["text"] for c in chunks])}

def search(index, question, k, only_status=None):
    """The k nearest chunks. The filter is applied BEFORE sorting: the excluded ones do not compete."""
    scores = index["M"] @ index["enc"].encode([question])[0]
    if only_status is not None:
        allowed = np.array([c["status"] == only_status for c in index["chunks"]])
        scores = np.where(allowed, scores, -np.inf)
    order = np.argsort(-scores, kind="stable")[:k]
    return [(index["chunks"][i], float(scores[i])) for i in order if np.isfinite(scores[i])]

Now the measure that gives the lesson its name. The function counts how many questions have the anchor inside one of the first k retrieved chunks.

def ceiling(retrieved, anchors, k):
    """Fraction of questions whose anchor is inside one of the first k chunks."""
    hits = 0
    for query_chunks, anchor in zip(retrieved, anchors):
        target = normalize(anchor)                      # whitespace collapsed on both sides:
        if any(target in normalize(c["text"])           # the anchor has the docstring's newlines,
               for c, _ in query_chunks[:k]):           # the chunk does not, and it is the same text
            hits += 1
    return hits / len(anchors)

def recall(index, k, only_status=None):
    return ceiling([search(index, q, k, only_status) for q in QUESTIONS], ANCHORS, k)

Ten lines: all the code you need to know, before writing a prompt, whether the problem is in retrieval. The normalisation of whitespace on both sides is not pedantry: the anchor has the docstring's newlines, the chunk does not, and without normalising the ceiling comes out close to zero and you hunt for an afternoon for a fault in retrieval that is in the string comparison. The first result, with 160-word windows overlapping by 32 and no filter:

ix = index_docs(documents, lambda t: fixed_window(t, 160, 32))
retrieved = [search(ix, q, 50) for q in QUESTIONS]
for k in (1, 3, 5, 10, 50):
    print(f"k={k:>2}  ceiling = {ceiling(retrieved, ANCHORS, k):.3f}")

k= 1  ceiling = 0.389
k= 3  ceiling = 1.000
k= 5  ceiling = 1.000
k=10  ceiling = 1.000
k=50  ceiling = 1.000

The ceiling at k = 3 is 1.000: the answer is always among the first three chunks. At k = 1 it is 0.389: the right chunk is first fewer than four times in ten. From the distance between the two numbers you read a precise diagnosis, that no prompt changes: the retriever finds the answer, it just puts it in the wrong place. Why, we will see shortly: in the first places there are also the contradictory notes, which talk about the same words. A generator that read only the first chunk would answer well fewer than four times in ten, whatever its prompt.

Segmentation: three strategies, chosen with an ablation

Cutting a document into chunks is the most underrated decision in the pipeline, because it happens once and produces no errors. Three strategies cover almost all cases, and their code is in the previous section. Here the unit is the word, not the token: there is no real tokeniser, and the proportion between the two depends on the model (the lesson on tokenisation measures how much).

They are not chosen by intuition: they are measured. Ablation on the corpus of 28 documents, with the fake encoder and no filter. The code runs four configurations; the columns are how many chunks come out, how many words on average, and what the recall is (recall@k: the fraction of questions with the anchor in the first k).

print(f"{'strategy':<34}{'chunks':>6}{'words/chunk':>12}{'recall@1':>9}{'recall@3':>9}")
for name, cut in [("fixed window 160, overlap 0", lambda t: fixed_window(t, 160, 0)),
                  ("fixed window 160, overlap 32", lambda t: fixed_window(t, 160, 32)),
                  ("by structure (paragraphs, max 160)", lambda t: by_structure(t, 160)),
                  ("by sentence (4 sentences, 1 overlap)", lambda t: by_sentence(t, 4, 1))]:
    i2 = index_docs(documents, cut)
    lengths = [len(c["text"].split()) for c in i2["chunks"]]
    print(f"{name:<36}{len(lengths):>4}{sum(lengths) / len(lengths):>12.1f}{recall(i2, 1):>9.3f}{recall(i2, 3):>9.3f}")

strategy                          chunks words/chunk recall@1 recall@3
fixed window 160, overlap 0           96       133.6    0.333    0.944
fixed window 160, overlap 32         116       134.7    0.389    1.000
by structure (paragraphs, max 160)   104       123.3    0.444    1.000
by sentence (4 sentences, 1 overlap) 343        49.9    0.389    0.889

The honest reading is that at k = 3 the first three strategies are almost level (0.944, 1.000, 1.000) and that the difference between 0.944 and 1.000 is one question in eighteen. The one by sentence, with 343 chunks against 96-116, is the only one that really loses (0.889) and it costs three times the index: fifty-word pieces do not carry the surroundings a term needs to tell one function from another. At k = 1 the values move between 0.333 and 0.444 but it is an even weaker signal, for the same reason. The rule to take away: if the documents have a structure, use it, because at equal recall it costs nothing; but do not believe a ranking between two configurations that differ by one question.

The second ablation separates the chunk size from the overlap, with the strategy held constant (fixed window, 20% overlap).

print(f"{'size':>5}{'ovl.':>6}{'chunks':>7}{'rec@1':>7}{'rec@3':>7}{'rec@5':>7}{'rec@50':>8}")
for size, ovl in [(40, 0), (40, 8), (80, 0), (80, 16), (160, 0), (160, 32), (320, 0), (320, 64)]:
    i2 = index_docs(documents, lambda t: fixed_window(t, size, ovl))
    print(f"{size:>5}{ovl:>6}{len(i2['chunks']):>7}{recall(i2, 1):>7.3f}{recall(i2, 3):>7.3f}"
          f"{recall(i2, 5):>7.3f}{recall(i2, 50):>8.3f}")

 size  ovl. chunks  rec@1  rec@3  rec@5  rec@50
   40     0    334  0.222  0.611  0.722   0.833
   40     8    413  0.222  0.611  0.778   0.889
   80     0    175  0.389  0.889  0.944   1.000
   80    16    214  0.333  0.944  1.000   1.000
  160     0     96  0.333  0.944  1.000   1.000
  160    32    116  0.389  1.000  1.000   1.000
  320     0     58  0.444  0.944  1.000   1.000
  320    64     67  0.389  1.000  1.000   1.000

Two readings. The first is that in this corpus small chunks lose: at 40 words the recall@5 is 0.722-0.778 and the recall@50 stops at 0.833-0.889, that is in two or three questions out of eighteen the answer does not exist in any retrievable chunk, because the anchor has been broken. The reason is the length of the documents: the median is 230 words, and a 40-word chunk contains neither the name of the function nor its value. The result is the reverse of what you often read about corpora of web pages, where large chunks dilute the vector, and it reminds you that the right size depends on the length and density of your documents: it must be measured, not copied. The second reading is the overlap: at 80 and 160 words it raises the recall@3 (0.889 to 0.944, 0.944 to 1.000), and at 40 it is not enough. The reason is in the next section.

The chunk that cuts a definition in half

There is a failure of the fixed window that does not show in any average, because it hits few questions and hits them totally. If the boundary between two chunks falls inside the sentence that contains the answer, that sentence no longer exists whole anywhere: half in chunk i, half in chunk i+1. And the index is built without an error and without a warning.

You measure it by sweeping the window sizes from 20 to 400 words (twenty values) and counting the anchors that do not fit whole in any chunk of their document: 18 anchors for 20 sizes, 360 trials, without overlap and with an overlap of 20% of the window. The code also prints the first concrete cut it finds from 60 words up.

def cut_anchors(size, ovl):
    n = 0
    for _, module, anchor in GOLDEN:
        pieces = fixed_window(by_id[module]["text"], size, ovl)
        n += not any(normalize(anchor) in normalize(p) for p in pieces)
    return n
def n_chunks(size, ovl):
    return sum(len(fixed_window(d["text"], size, ovl)) for d in documents)
sizes = range(20, 401, 20)
without = [cut_anchors(s, 0) for s in sizes]
with_ovl = [cut_anchors(s, s // 5) for s in sizes]
print(f"{'size':>5}{'cut without ovl.':>18}{'cut with 20%':>14}")
for s, a, b in zip(sizes, without, with_ovl):
    if a or b:
        print(f"{s:>5}{a:>18}{b:>14}")
print(f"total trials: {len(sizes)} sizes x {len(GOLDEN)} anchors = {len(sizes) * len(GOLDEN)}")
print(f"anchors cut WITHOUT overlap: {sum(without)}/{len(sizes) * len(GOLDEN)}")
print(f"anchors cut WITH a 20% overlap: {sum(with_ovl)}/{len(sizes) * len(GOLDEN)}")
c0 = sum(n_chunks(s, 0) for s in sizes); c1 = sum(n_chunks(s, s // 5) for s in sizes)
print(f"cost of the overlap: {c1} chunks against {c0} = +{100 * (c1 - c0) / c0:.1f}% of index")
def first_cut(min_size):
    for s in sizes:
        if s < min_size:
            continue
        for _, module, anchor in GOLDEN:
            doc_words = by_id[module]["text"].split()
            a = normalize(anchor).split()
            for i in range(len(doc_words) - len(a) + 1):
                if doc_words[i:i + len(a)] == a:
                    b = (i // s + 1) * s                    # first boundary after the start
                    if i < b < i + len(a):
                        return s, module, anchor, doc_words[b - 6:b], doc_words[b:b + 6]
s, module, anchor, tail, head = first_cut(60)
print(f"example at size {s}, module {module}, anchor: {anchor!r}")
print("  tail of chunk i  :", "..." + " ".join(tail))
print("  head of chunk i+1:", " ".join(head) + " ...")

 size  cut without ovl.  cut with 20%
   20                 3             4
   40                 3             2
   60                 1             0
  100                 1             0
  120                 1             0
  200                 1             0
  260                 1             0
total trials: 20 sizes x 18 anchors = 360
anchors cut WITHOUT overlap: 11/360
anchors cut WITH a 20% overlap: 6/360
cost of the overlap: 3215 chunks against 2646 = +21.5% of index
example at size 60, module difflib, anchor: 'Optional arg cutoff (default 0.6)'
  tail of chunk i  : ...> 0. Optional arg cutoff (default
  head of chunk i+1: 0.6) is a float in [0, ...

Without overlap 11 anchors out of 360 are cut (3.1%); with 20%, 6 are cut (1.7%), and all six are at 20 and 40 words. Above 60 words the overlap eliminates the cuts, but below it does not, and the reason is a criterion you can compute: the overlap must be at least as long as the longest anchor you want to protect. At 20 words the overlap is 4, at 40 it is 8, and an anchor like "Optional arg cutoff (default 0.6)" has six: its sentence can still fall across the boundary. The cost is an index larger by 21.5% (3215 chunks against 2646, summing the twenty sizes), and the same sentence, now present in two chunks, can occupy two of the k positions, a side effect to take into account when choosing k.

The printed example is concrete: at 60 words the name of the parameter, cutoff, is at the end of one chunk and its value, 0.6, at the head of the next. Neither answers, and the second does not contain the word cutoff, so it is not even retrieved for that question. A handful of questions that will never work, with any prompt, while the average is fine: this is the kind of failure that only a golden set with literal anchors makes visible.

Metadata and filters: the errors you avoid without hunting for them

Part of the errors of a RAG have nothing to do with semantic similarity. A document not visible to that user, a superseded version, a draft, an expired document: all are relevant to the question, so retrieval by similarity ranks them high - and it is the correct behaviour of a system that measures similarity. The way to exclude them is not to teach the vector to tell them apart: it is not to put them among the candidates.

That is why metadata go in the index, next to the vector, and the filter is applied before the search. Real vector indexes provide for it: the Redis documentation describes vector searches with filters "to narrow the search space" on text, numeric, geographic and TAG fields. The shape of the request is this (a sketch to adapt, not executed: there is no Redis listening here):

# sketch to adapt, not executed (syntax from the Redis documentation)
FT.CREATE idx:documents ON HASH PREFIX 1 doc: SCHEMA
    status TAG
    text TEXT
    vector VECTOR HNSW 6 TYPE FLOAT32 DIM 1024 DISTANCE_METRIC COSINE

FT.SEARCH idx:documents "(@status:{current})=>[KNN 3 @vector $q AS distance]"
    PARAMS 2 q <vector-blob> DIALECT 2

The field is a TAG and not a TEXT because it is an exact value ("current" or "draft"), not a text to tokenise. In our code the same filter is the only_status parameter of search. The block that follows measures what it is worth, with the contradictory notes in the corpus:

ix = index_docs(documents, lambda t: fixed_window(t, 160, 32))
excluded = sum(c["status"] != "current" for c in ix["chunks"])
print(f"chunks excluded by the filter: {excluded}/{len(ix['chunks'])} ({100 * excluded / len(ix['chunks']):.2f}%)")
for label, f in (("no filter", None), ("status = current", "current")):
    top3 = [search(ix, q, 3, f) for q in QUESTIONS]
    contradicted = sum(any(c["status"] != "current" for c, _ in t) for t in top3)
    print(f"{label:<18} recall@1 {recall(ix, 1, f):.3f} | recall@3 {recall(ix, 3, f):.3f} | "
          f"questions with a contradictory chunk in the top 3: {contradicted}/{len(QUESTIONS)}")

chunks excluded by the filter: 10/116 (8.62%)
no filter          recall@1 0.389 | recall@3 1.000 | questions with a contradictory chunk in the top 3: 10/18
status = current   recall@1 0.889 | recall@3 1.000 | questions with a contradictory chunk in the top 3: 0/18

Ten chunks out of 116 (8.62% of the index) are superseded notes. Without the filter, in 10 questions out of 18 at least one of the three chunks handed to the model states the opposite of the library; with the filter, zero. The recall@3 does not change (it stays 1.000) but the recall@1 rises from 0.389 to 0.889: the contradictory chunks share the words of the question, so they are often on top, and the filter is the move that fixes the ordering that earlier looked like a problem of the retriever. The rule to take away: the metadata filter eliminates whole sets of errors at almost no cost. Doing it downstream does not work: you have already consumed the k positions with the documents you wanted to exclude. The flip side is that a filtered chunk no longer exists for any question: if the field is wrong, or if the draft was the only document that answered, recall drops, and the symptom is "it never finds that answer".

Reranking: two independent passes against one pass over the pair

A vector index uses a bi-encoder: the document becomes a vector when you index it, the question when it arrives, and the two transformations are independent. It is what makes the index possible, and at the same time its limit: when the document becomes a vector the question does not exist yet, so the compression cannot know which part of the document will be the relevant one.

A cross-encoder does the opposite: it takes the pair (question, chunk) and processes it in a single pass, with the terms of the question and those of the document looking at each other. It is more accurate and cannot be indexed, because it has to be run on every pair at question time (Nogueira and Cho, Passage Re-ranking with BERT, arXiv:1901.04085). Hence the two-stage scheme: the bi-encoder retrieves wide, the cross-encoder reranks narrow.

There is no neural cross-encoder here, and I replace it with a lexical surrogate, declared as such: a joint score that looks at the coverage of the question's terms, shared bigrams and the proximity of the terms inside the chunk. It is not a trained model, but it is a computation on the pair, which is the property to understand. The weights 0.55 / 0.30 / 0.15 are my choice, not optimised.

def rerank_lexical(query, candidates):
    """Cross-encoder surrogate: ONE pass over the PAIR (query, chunk). It is not a model."""
    qt = words(query)                             # question terms, without stopwords
    qbi = set(zip(qt, qt[1:]))                    # bigrams of the question
    out = []
    for c, sim in candidates:
        ct = words(c["text"])
        pos = {}
        for i, w in enumerate(ct):
            pos.setdefault(w, []).append(i)
        found = [w for w in qt if w in pos]
        coverage = len(set(found)) / len(set(qt))
        bigrams = len(qbi & set(zip(ct, ct[1:]))) / max(1, len(qbi))
        if len(found) >= 2:                       # are the question terms close together
            indices = sorted(pos[w][0] for w in set(found))   # in the chunk, or scattered?
            proximity = len(indices) / (indices[-1] - indices[0] + 1)
        else:
            proximity = 0.0
        out.append((c, 0.55 * coverage + 0.30 * bigrams + 0.15 * proximity, sim))
    out.sort(key=lambda x: (-x[1], -x[2]))        # on equal score the bi-encoder decides
    return [(c, s) for c, s, _ in out]

def retrieve(index, question, k_candidates=50, k_final=5, rerank=True, only_status=None):
    candidates = search(index, question, k_candidates, only_status)
    if rerank:
        candidates = rerank_lexical(question, candidates)
    return candidates[:k_final]

Retrieval becomes: search k = 50 candidates with the bi-encoder, rerank, keep the first few. The block compares three chunk sizes, always with the filter on, and then times the two routes: for each configuration it computes the recall of the bi-encoder alone and the one after reranking, at k = 1 and k = 3, plus the recall@50 that acts as the ceiling of the candidates. The timer repeats every question twenty times and reports the minimum and the maximum of three measurements.

print(f"{'chunk':>10}{'rec@50':>8}{'rec@1 bi':>10}{'rec@1 rerank':>14}{'rec@3 bi':>10}{'rec@3 rerank':>14}")
for size, ovl in [(40, 8), (80, 16), (160, 32)]:
    i2 = index_docs(documents, lambda t: fixed_window(t, size, ovl))
    bi = lambda k: ceiling([search(i2, q, k, "current") for q in QUESTIONS], ANCHORS, k)
    rr = lambda k: ceiling([retrieve(i2, q, 50, k, True, "current") for q in QUESTIONS], ANCHORS, k)
    print(f"{size:>6}/{ovl:<3}{bi(50):>8.3f}{bi(1):>10.3f}{rr(1):>14.3f}{bi(3):>10.3f}{rr(3):>14.3f}")
def ms_per_question(f, repetitions=20):
    t0 = time.perf_counter()
    for _ in range(repetitions):
        for q in QUESTIONS:
            f(q)
    return 1000 * (time.perf_counter() - t0) / (repetitions * len(QUESTIONS))
bi_only = [ms_per_question(lambda q: search(ix, q, 50, "current")) for _ in range(3)]
with_rr = [ms_per_question(lambda q: retrieve(ix, q, 50, 3, True, "current")) for _ in range(3)]
print(f"latency per question, 3 runs: bi-encoder only {min(bi_only):.2f}-{max(bi_only):.2f} ms | "
      f"with reranking {min(with_rr):.2f}-{max(with_rr):.2f} ms")

     chunk  rec@50  rec@1 bi  rec@1 rerank  rec@3 bi  rec@3 rerank
    40/8     0.889     0.278         0.500     0.667         0.667
    80/16    1.000     0.722         0.889     0.944         0.944
   160/32    1.000     0.889         0.944     1.000         1.000
latency per question, 3 runs: bi-encoder only 0.57-0.65 ms | with reranking 5.39-6.87 ms

Reranking helps where the bi-encoder is weak and not where it is already strong. With 40-word chunks the recall@1 goes from 0.278 to 0.500, at 80 from 0.722 to 0.889, at 160 from 0.889 to 0.944: the more retrieval is in difficulty, the more reranking gains. But the recall@3 never moves (0.667, 0.944, 1.000 before and after), and the first column explains why: at 40 words the recall@50 is 0.889, so in two questions the answer is not among the candidates and no reranking can invent it. It is the general rule: reranking moves up what the candidates already contain. If the recall@50 is low, the fault is upstream, in the segmentation.

The cost is measured in milliseconds. Over six runs of the program, on a shared machine, the bi-encoder search cost 0.24-0.73 ms per question and the search plus reranking 5.1-7.9 ms (the line printed above is only one of the six). That is ten milliseconds or less, nothing compared with the seconds of a real generation, and for that reason reranking almost always pays when there is a gain. There is one important condition: our surrogate looks at the same words as the encoder, so it brings no new information and gains little. A real cross-encoder brings a different signal, and a reranking that repeats the retrieval signal costs time and gives nothing.

The grounding prompt: three requirements written as requirements

The prompt has a narrow task and three requirements, to be written as requirements and not as good intentions. First: answer only from the context, spelling out the hard case - use nothing that the model knows from other sources, even if it is sure it is true. Second: declare not knowing with an exact sentence, because "say you don't know" is an invitation and a fixed string is a contract that can be checked downstream. Third: cite, sentence by sentence, the identifier of the span used.

INSTRUCTIONS = """You are an assistant that answers ONLY with what is written in the context.

Rules, all three must be respected:
1. Use only the information in the spans below. Do not use anything
   you know from other sources, even if you are certain it is true.
2. If the spans do not contain the answer, write exactly this line and nothing else:
   NOT IN THE CONTEXT
3. Every sentence of the answer must end with the identifier of the span you
   took it from, in square brackets, like this: [S3]. A sentence without an
   identifier is a violation. Do not cite spans you did not use.

Answer in English, in at most two sentences."""

def build_prompt(question, span_texts):
    context = "\n\n".join("[S%d] %s" % (i + 1, t) for i, t in enumerate(span_texts))
    return INSTRUCTIONS + "\n\nSPANS:\n\n" + context + "\n\nQUESTION: " + question + "\n\nANSWER:"

The function numbers the spans from [S1] onward, because the identifier is what will let the verifier go back to the text. How long the built prompt is I measure further down, with the three spans of the first question. The call to a local model is a POST and nothing more; I give it as a sketch because there is no model listening here and I did not run it.

# sketch to adapt, not executed: needs a local runtime listening and a downloaded model
import json, urllib.request

def call(prompt, model, num_predict=200, seed=7):
    req = urllib.request.Request(
        "http://localhost:11434/api/generate",
        data=json.dumps({"model": model, "prompt": prompt, "stream": False,
                         "options": {"temperature": 0, "seed": seed,
                                     "num_predict": num_predict,
                                     "num_ctx": 16384}}).encode(),   # always declare it
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=1800) as r:
        return json.load(r)["response"].strip()

The options are not details. temperature: 0 with a fixed seed makes the experiment repeatable; num_predict limits the length of the answer; num_ctx is, according to the Ollama API documentation, the "context length size (number of tokens)". Declaring it explicitly is prudence: a prompt longer than the runtime's window cannot fit in it entirely, and I chose not to leave the value implicit because I have not verified what the exact behaviour of your runtime is in that case. The output format is minimal - prose with [Sn] markers - because the aim is traceability. When the answer has to go into a program you need a validated schema: that is the subject of the lesson Structured Output: Getting Data You Can Validate Out of an Agent. The two mechanisms compose: schema for the shape, citation for the provenance.

The verification that does not trust

Asking for the citation is not enough. A model can cite a span that does not say that thing, or produce a plausible number and attach to it an equally plausible [S2]. The citation, by itself, is a statement by the model about its own behaviour: the one category of statement you cannot accept from a probabilistic system.

You need a programmatic check: for each sentence go back to the cited span and demand that every number and every technical name appear literally in there. The rule is crude and for that reason robust: it does not interpret meaning, it compares strings.

CITATION = re.compile(r"\[S(\d+)\]")
NUMBER = re.compile(r"-?\d+(?:\.\d+)?")
# A technical name has a dot, an underscore or an inner capital.
# Common English words have none: that is how "According" at the start
# of a sentence does not get a correct answer refused.
NAME = re.compile(r"\b(?:[A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*"
                  r"|[A-Za-z]*[a-z][A-Z][A-Za-z]*"
                  r"|[a-z]+_[a-z_]+)\b")

def verify_grounding(answer, spans):
    if answer.strip().startswith("NOT IN THE CONTEXT"):
        return "abstained"
    sentences = [f.strip() for f in re.split(r"(?<=[.!?])\s+", answer) if f.strip()]
    if not sentences:
        return "refused"
    for sentence in sentences:
        cited = CITATION.findall(sentence)
        if not cited:
            return "refused"                       # a sentence with no provenance
        texts = []
        for c in cited:
            n = int(c)
            if not (1 <= n <= len(spans)):
                return "refused"                   # it cited a span that does not exist
            texts.append(spans[n - 1])
        union = " ".join(texts).lower()
        bare = CITATION.sub("", sentence)          # mandatory: without this, the "3"
        for number in NUMBER.findall(bare):        # of "[S3]" would be searched in the span
            if number.lower() not in union:
                return "refused"
        for name in NAME.findall(bare):
            if name.lower() not in union:
                return "refused"
    return "anchored"

Three choices in the code deserve a line. The number of the identifier is removed from the sentence before looking for numbers (otherwise the 3 of [S3] would be searched in the span). The rule for technical names - a dot, an underscore or an inner capital - serves to avoid refusing a correct answer because it starts with "According". And an identifier outside the range ([S4] with three spans) is a violation, not an error to ignore. The six answers that follow are written by me, not produced by a model, and cover the cases:

context = ["Optional arg cutoff (default 0.6) is a float in [0, 1].",
           "NOTE (draft): the default cutoff of get_close_matches is 0.9."]
cases = ["The default cutoff is 0.6.",
         "The default cutoff is 0.9 [S1].",
         "The default cutoff is 0.9 [S2].",
         "The default cutoff is 0.6 [S1].",
         "The default cutoff is 0.6 [S4].",
         "NOT IN THE CONTEXT"]
for r in cases:
    print(f"{verify_grounding(r, context):<10} <- {r}")

refused    <- The default cutoff is 0.6.
refused    <- The default cutoff is 0.9 [S1].
anchored   <- The default cutoff is 0.9 [S2].
anchored   <- The default cutoff is 0.6 [S1].
refused    <- The default cutoff is 0.6 [S4].
abstained  <- NOT IN THE CONTEXT

The cases read like a truth table. Without a citation, refused: even if the content is right, because the verifier evaluates traceability and not truth. With 0.9 cited on [S1], refused: it is the case the verifier exists for, an invented number with a plausible citation. With 0.6 cited on [S1], anchored. With [S4], refused. And with 0.9 on [S2], anchored - and false, because the span [S2] is the draft note. Answers that do not pass are not shown with a warning: they are refused, and the percentage of refusals becomes an operating metric. A system with zero refusals is not perfect, it is unchecked.

The check guarantees that the answer is anchored, not that it is true. If a wrong document ended up in the context, a faithfully wrong answer passes with full marks. Against that you need the metadata filter, not the verifier: two distinct defences for two distinct failures. To see it on all eighteen questions I put the whole pipeline together, with the scripted generator in place of the model:

def fake_model(question, spans):
    """Scripted generator: it is NOT a model. It takes the sentence of the spans that shares
    the most terms with the question and cites it; if it shares fewer than two, it abstains."""
    qt = set(words(question))
    points, span_no, chosen = 0, 0, ""
    for n, text in enumerate(spans, 1):
        for sentence in re.split(r"(?<=[.!?])\s+", normalize(text)):
            p = len(qt & set(words(sentence)))
            if p > points:
                points, span_no, chosen = p, n, sentence
    if points < 2:
        return "NOT IN THE CONTEXT"
    return f"{chosen.rstrip('.')} [S{span_no}]."

def run_pipeline(index, only_status=None, k_final=3):
    outcomes = collections.Counter()
    for (question, module, anchor) in GOLDEN:
        top = retrieve(index, question, 20, k_final, True, only_status)
        spans = [c["text"] for c, _ in top]
        answer = fake_model(question, spans)
        outcome = verify_grounding(answer, spans)
        if outcome == "anchored":
            n = int(CITATION.findall(answer)[0])
            outcome += " / right source" if top[n - 1][0]["doc"] == module else " / WRONG source"
        outcomes[outcome] += 1
    return outcomes

Before putting everything together, the real prompt. The block builds the request for the first question of the golden set, with the first three spans reranked and filtered, and shows how long it is and what the scripted generator would answer:

q = GOLDEN[0][0]
top = retrieve(ix, q, 20, 3, True, "current")
spans = [c["text"] for c, _ in top]
prompt = build_prompt(q, spans)
print(f"prompt: {len(prompt.split())} words, {len(prompt)} characters; spans delivered: {len(spans)}")
print("answer of the scripted generator:", fake_model(q, spans))

prompt: 607 words, 3861 characters; spans delivered: 3
answer of the scripted generator: Function get_close_matches(word, possibilities, n=3, cutoff=0.6): Use SequenceMatcher to return list of the best "good enough" matches [S3].

607 words for three spans: the model's window has to be sized on this number multiplied by the words-to-tokens factor of your model, not on what you hope it is. The scripted generator's answer is the header sentence of the function, with the citation [S3]: it contains the name and the signature, with cutoff=0.6 inside, but not the sentence that explains the parameter. It is the limit of a generator built on word overlap, and worth remembering when reading the table that follows.

for label, f in (("no filter", None), ("status = current", "current")):
    outcomes = run_pipeline(ix, f)
    print(f"{label:<18}", dict(sorted(outcomes.items())))

no filter          {'anchored / WRONG source': 10, 'anchored / right source': 8}
status = current   {'anchored / right source': 18}

Without the filter, the scripted generator cites in 10 questions out of 18 a span that comes from a note and not from the right module, and the verifier gives all ten as anchored: they cite, the numbers are there, no invented word. With the filter, all eighteen come from the right module. It is the measured demonstration of the previous paragraph, and the "ten" coincides with the number of questions that had a contradictory chunk in the top three. Be careful about what it does not say: "right source" means that the cited span comes from the expected module, not that the chosen sentence contains the answer; the scripted generator chooses by word overlap and, for the first question, chose the header sentence of the function (see above) and not the one about cutoff. A real generator does better, or worse, in ways that I do not measure here.

[S1] Optional arg cutoff (default 0.6) is a float in [0, 1].
[S2] NOTE (draft): the default cutoff of get_close_matches is 0.9.

Trade-offs

• Retrieval against generation. The generator does not repair a wrong retrieval, and it is not a prompt defect: it is a limit. Here the ceiling at k = 3 is 1.000 but at k = 1 it is 0.389: the distance is all ordering. Mitigation: measure the ceiling at wide k and at narrow k before touching the prompt; the distance tells you which component you are about to work on.
• Small chunks against large chunks. On this corpus at 40 words the recall@50 stops at 0.889 because the anchors break, at 160 it reaches 1.000. The price of the large one is a vector that averages several topics, which does not emerge here because the encoder is lexical and the documents are short. Mitigation: measure the size on your documents, starting from a value close to the length of the typical paragraph.
• Segmentation by structure against segmentation by sentence. By structure 104 chunks and recall@3 1.000, by sentence 343 chunks and 0.889. Mitigation: if the documents have paragraph boundaries, use them; the fixed window serves when the text is an undifferentiated stream.
• Overlap against index size. A 20% overlap takes the cuts from 11 to 6 out of 360 trials and costs 21.5% more chunks; below 60 words the overlap is shorter than the anchor and does not protect. Mitigation: size the overlap on the length of the sentence to protect, not on a percentage.
• Reranking against latency. The surrogate takes the recall@1 from 0.278 to 0.500 with 40-word chunks, and the cost from under a millisecond to 5-8 ms. It pays where retrieval is weak and the candidates contain the answer; on recall@3 nothing ever changes. Mitigation: decide by looking at the recall@50 of the candidates, not at the reranker's promise.
• Metadata filter against recall. The filter excludes 10 chunks out of 116, takes the questions with a contradictory chunk from 10/18 to 0/18 and the recall@1 from 0.389 to 0.889. The flip side is that a filtered chunk no longer exists for any question. Mitigation: measure the recall@k with and without the filter.
• Mandatory citation against answer rate. The verifier refuses every sentence without provenance and every number that does not appear in the span: a correct answer that the model cannot justify ends up in there too. The gain is that nothing reaches the reader without a traceable source; the price is a refusal rate above zero, which with a real model has to be measured on your own golden set. Mitigation: keep the refusal rate as an operating metric, next to accuracy.

The seven items have a common structure: each choice buys something and pays for it elsewhere, and the mitigation is almost always a measurement, not a parameter to remember. Three of them (ceiling, filter, overlap) must be measured before any optimisation, because they change what the rest of the pipeline has to do.

The traps you really fall into

1. Rewriting the prompt before measuring the ceiling. Symptom: an afternoon of reworded instructions and accuracy that moves by a hair. Cause: the answer is not in the chunks handed over, or is at the bottom of the list. Remedy: the ten lines of ceiling(...) before anything else.
2. Reading the similarity score as a percentage. Symptom: "the first result is at 0.78, so it is 78% good", and a hand-written threshold that cuts away the right answers. Cause: an encoder's space is not calibrated, and the absolute value means nothing outside its distribution. Remedy: order metrics only, and the threshold derived from the percentiles of your corpus, redone at every model change.
3. Writing the evaluation questions while looking at the chunks. Symptom: excellent recall@k in evaluation and a system that finds nothing in production. Cause: the questions reuse the words of the chunk they must retrieve, so you measure the index's memory instead of retrieval. Here it is a concrete risk: my questions contain the names of the functions, and a lexical encoder loves that. Remedy: question phrased as a user would write it, anchor taken from the document, size of the set declared.
4. Laying out the spans without deciding the order. Symptom: the same set of chunks gives different answers depending on how you line them up, and the worst case is when the good one is in the middle. Cause: models use the beginning and the end of the context better than the centre (Liu and colleagues, Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172). Remedy: few spans, the best at the two extremes, and the order treated as a variable to measure - here I did not measure it, because the scripted generator has no middle of the context to get lost in.
5. Trusting the citation because it is there. Symptom: well-formed answers, an [Sn] after every sentence, and a wrong number inside. Cause: the citation, by itself, is a statement by the model about its own behaviour. Remedy: the literal comparison between the numbers of the sentence and the text of the span.
6. Letting the chunk boundary cut the sentence that answers. Symptom: a handful of questions that never work, with any prompt, while the average is fine. Cause: the sentence no longer exists whole anywhere. Remedy: an overlap longer than the anchor, knowing that the same sentence, now present in two chunks, can occupy two of the k positions.

Six different symptoms, a single habit that prevents them: measuring one piece at a time, with the ceiling as the first number, and always comparing before and after. Traps 1, 3 and 5 are those where a system seems to work because the question you see is the one you wrote yourself; the other three only show up with real documents and users.

Rerunning the experiment

All the blocks of this lesson are a single program: put them in a file in the order in which they appear (corpus, golden set, encoder, index, ceiling, chunkers, reranking, prompt, verifier, scripted generator, then the measurement blocks) and launch it with python3; you need NumPy 2.x and no other package, no network and no model. I ran it twice and the outputs, apart from the timing line, are identical. The exercises with the parts to complete are in esercizi.py, downloadable from the course repository: I did not run it for this revision, and its exercise 3b uses a local runtime that is not here.

What to take away

A RAG is an open-book exam, and almost all the work lies in the hand that opens the book: the retrieval built in the lesson on embeddings and semantic search. The number to look at before any other is the ceiling at wide k and at narrow k, not the final accuracy: here 1.000 at k = 3 against 0.389 at k = 1 says that the retriever always finds the answer and puts it in the wrong place, and it is the diagnosis that sends you to the filter and the reranking instead of the prompt. The rest are three measurable decisions: cut with an overlap longer than the anchor (from 11 to 6 cuts out of 360, zeroed above 60 words), filter on metadata before searching (from 10/18 questions with a contradictory document in the context to 0/18), and rerank only where the candidates already contain the answer.

The second thing to take away is that the citation is not an editorial courtesy: it is the one point where a probabilistic system becomes controllable by a program. The verifier compares strings and not meanings, and it works for exactly that reason, and for exactly that reason it guarantees grounding, not truth: the wrong answer cited correctly passes with full marks, and is repaired upstream with the filter. What remains open is the measurement with a real model: here encoder and generator are fake, 18 questions barely separate 0.389 from 0.889, and the refusal rate of a real model I have not measured. Sizing a golden set, choosing the metrics and using a model as a judge - with the biases that judge brings with it - is the craft of the next lesson, on evaluating generative systems.

The same series also includes Running a Model Locally: Quantisation, Memory and Latency, Measured.

Sources

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401) - the RAG architecture and its name; consulted on 2026-11-12.
• Nogueira and Cho, Passage Re-ranking with BERT (arXiv:1901.04085) - reranking with a model that reads the question-passage pair; consulted on 2026-11-12.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172) - model performance depends on the position of the information in the context, higher at the beginning and the end; consulted on 2026-11-12.
• Redis, vector search documentation - vector searches with filters on TAG, text and numeric fields, and the syntax of FT.SEARCH with KNN; consulted on 2026-11-12.
• Ollama, generate API - the num_ctx option as the context length in tokens; consulted on 2026-11-12.
• Python, difflib documentation - the default value cutoff=0.6 of get_close_matches, used as the golden-set answer; consulted on 2026-11-12.
• The figures in the lesson come from the listings printed here, run twice with identical output (the timings: six runs, reported as a range). No model was run: encoder and generator are fake and declared in the text.

The first six sources are primary and I opened them on 2026-11-12 to check the title and the content cited; the last entry instead declares where the figures come from, because a measurement made in the repository is cited with the command that produces it.