08 - Prompt optimization driven by eval: A/B retrieval strategy, reranking, prompt template

Optimizing RAG with evaluation: one variable at a time, a paired test

You change the prompt because it "sounds better", add a reranker because you read good things about it, switch from BM25 to a vector index: the system probably improves, but you don't know by how much, or which of the three moves made the difference. Optimizing by eye produces improvements by chance and regressions by chance. The remedy is the same as for any experiment: one variable at a time, a fixed evaluation set, the same question asked of both systems, and a test that says whether the difference is larger than the noise.

Here you build that test bench with numbers you can redo. The lab is deliberately small and synthetic (forty documents, eighty queries) to show the mechanism; the values that come out are not to be read as the performance of a real system, but as an illustration of what confidence interval, correction for multiple comparisons and segmentation mean. The retrieval metrics used are those of episode 5; for hybrid retrieval itself see this deep dive from the AI Engineering series, and for the reranker the article dedicated to re-ranking.

The test bench: three systems, one metric per query

The corpus has forty guides, one for each (action, object) pair, such as "to configure the cache open the panel...". There are eighty queries: forty with the verb form present in the document ("configure cache"), forty with the noun that the document does not contain ("configuration cache"). You need three retrievers. A is BM25 written in numpy, which compares whole words. B is TF-IDF on character n-grams (three to five) with scikit-learn, which forgives morphological variations. C merges the two rankings with Reciprocal Rank Fusion, which adds 1/(k + rank) for each ranking with k = 60. The metric is the reciprocal of the rank of the right document, computed for each query: it is this vector, and not just the mean, that the test needs.

import math
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

ACTIONS = [("configure", "configuration"), ("restart", "restarting"), ("monitor", "monitoring"),
           ("update", "updating"), ("delete", "deletion"), ("import", "importing"),
           ("export", "exporting"), ("protect", "protection")]
OBJECTS = ["cache", "database", "log", "certificate", "backup"]

# 40 documents, one per (action, object) pair; 80 queries: half with the same form
# as the document, half with the noun ("configuration cache") that the text does not contain.
DOCS = [f"Guide: to {verb} the {obj} open the panel and follow procedure {i}."
        for i, (verb, noun) in enumerate(ACTIONS) for obj in OBJECTS]
RIGHT = list(range(len(DOCS)))
Q_EXACT = [f"{verb} {obj}" for verb, noun in ACTIONS for obj in OBJECTS]
Q_MORPH = [f"{noun} {obj}" for verb, noun in ACTIONS for obj in OBJECTS]
QUERIES = Q_EXACT + Q_MORPH
TARGET = RIGHT + RIGHT
SEGMENT = ["exact"] * len(Q_EXACT) + ["morphological"] * len(Q_MORPH)


def bm25(corpus, k1=1.5, b=0.75):
    tok = [d.lower().replace(".", "").split() for d in corpus]
    vocab = sorted({t for d in tok for t in d})
    idx = {t: i for i, t in enumerate(vocab)}
    tf = np.zeros((len(corpus), len(vocab)))
    for r, d in enumerate(tok):
        for t in d:
            tf[r, idx[t]] += 1
    length = tf.sum(1)
    df = (tf > 0).sum(0)
    idf = np.log(1 + (len(corpus) - df + 0.5) / (df + 0.5))
    norm = tf * (k1 + 1) / (tf + k1 * (1 - b + b * length[:, None] / length.mean()))

    def scores(q):
        v = np.zeros(len(vocab))
        for t in q.lower().split():
            if t in idx:
                v[idx[t]] = 1
        return (norm * idf) @ v
    return scores


def char_tfidf(corpus):
    vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5)).fit(corpus)
    m = vec.transform(corpus)
    return lambda q: (m @ vec.transform([q]).T).toarray().ravel()


def rrf(*scores, k=60):
    """Reciprocal Rank Fusion: sum 1/(k + rank) over several rankings."""
    total = np.zeros(len(scores[0]))
    for p in scores:
        rank = np.empty(len(p), dtype=int)
        rank[np.argsort(-p, kind="stable")] = np.arange(1, len(p) + 1)
        total += 1 / (k + rank)
    return total


def rr_per_query(function):
    """Reciprocal rank of the right document, one figure per query."""
    out = []
    for q, t in zip(QUERIES, TARGET):
        order = np.argsort(-function(q), kind="stable")
        out.append(1 / (int(np.where(order == t)[0][0]) + 1))
    return np.array(out)


A, B = bm25(DOCS), char_tfidf(DOCS)
SYSTEMS = {"A bm25": A, "B char-ngram": B, "C rrf(A,B)": lambda q: rrf(A(q), B(q))}
RR = {name: rr_per_query(f) for name, f in SYSTEMS.items()}

if __name__ == "__main__":
    seg = np.array(SEGMENT)
    print(f"{len(DOCS)} documents, {len(QUERIES)} queries")
    for name, rr in RR.items():
        print(f"{name:13s} MRR={rr.mean():.3f}  exact={rr[seg == 'exact'].mean():.3f}"
              f"  morphological={rr[seg == 'morphological'].mean():.3f}")

# Output:
# 40 documents, 80 queries
# A bm25        MRR=0.670  exact=1.000  morphological=0.340
# B char-ngram  MRR=1.000  exact=1.000  morphological=1.000
# C rrf(A,B)    MRR=0.969  exact=1.000  morphological=0.938

Read by segment, the figures say more than the mean. On the "exact" queries all the systems are perfect (1.000): comparing on those alone you would see no difference at all. On the morphological ones BM25 drops to 0.340, because the noun does not appear in the document and the only word that matches is the object, common to eight guides. The character n-grams reach 1.000 because "configur" is a shared stem. The fusion C sits in between (0.938): the BM25 branch brings noise into a branch that is already perfect. Be careful not to generalize: this is a fact about the constructed corpus, not a law of hybrid retrieval, which on real data often helps precisely because the two branches fail on different queries.

The paired test: is the difference larger than the noise?

The three columns are measured on the same queries, so they must be compared in pairs: for each query the difference between the two systems, then a test on the differences. The Wilcoxon signed-rank test (scipy.stats.wilcoxon) is non-parametric: its null hypothesis is that the distribution of the differences is symmetric around zero, and it doesn't require the metrics to be normal, which is false for scores bounded between 0 and 1 and often concentrated at the two extremes. With the default setting (zero_method="wilcox") the pairs with zero difference are discarded, which is why the listing prints how many there are. Next to the p-value it computes a 95% confidence interval for the mean difference with the bootstrap (fixed seed), because the p-value says only whether, not by how much.

import numpy as np
from scipy.stats import wilcoxon
from a_retrieval import RR


def holm(pvalues):
    """Holm correction for multiple comparisons: corrected p in the same order as the input."""
    order = np.argsort(pvalues)
    m = len(pvalues)
    corrected = np.empty(m)
    running_max = 0.0
    for pos, i in enumerate(order):
        running_max = max(running_max, min(1.0, (m - pos) * pvalues[i]))
        corrected[i] = running_max
    return corrected


def bootstrap_ci(diff, n=5000, seed=11):
    rng = np.random.default_rng(seed)
    means = [rng.choice(diff, size=len(diff), replace=True).mean() for _ in range(n)]
    return np.percentile(means, [2.5, 97.5])


comparisons = [("B", "A bm25"), ("C", "A bm25"), ("C", "B char-ngram")]
names = {"B": "B char-ngram", "C": "C rrf(A,B)"}
rows, raw_p = [], []
for x, y in comparisons:
    diff = RR[names[x]] - RR[y]
    zeros = int((diff == 0).sum())
    p = wilcoxon(diff).pvalue          # paired: same query in both arms
    lo, hi = bootstrap_ci(diff)
    rows.append((names[x], y, diff.mean(), lo, hi, zeros))
    raw_p.append(p)
for r, p, pc in zip(rows, raw_p, holm(np.array(raw_p))):
    print(f"{r[0]} - {r[1]}: delta={r[2]:+.3f} CI95%[{r[3]:+.3f}, {r[4]:+.3f}] "
          f"unchanged queries={r[5]}  p={p:.2g}  p_Holm={pc:.2g}")

# Output:
# B char-ngram - A bm25: delta=+0.330 CI95%[+0.249, +0.416] unchanged queries=45  p=2.3e-07  p_Holm=7e-07
# C rrf(A,B) - A bm25: delta=+0.299 CI95%[+0.222, +0.381] unchanged queries=45  p=2.4e-07  p_Holm=7e-07
# C rrf(A,B) - B char-ngram: delta=-0.031 CI95%[-0.066, -0.006] unchanged queries=76  p=0.063  p_Holm=0.063

There are three things to read. B against A: mean difference +0.330, interval from +0.249 to +0.416, p of the order of 10^-7: on 45 queries the result doesn't change and on the other 35 B wins. C against B: the difference is negative (-0.031) and the bootstrap interval does not include zero, yet the Wilcoxon p-value is 0.063, above 0.05. The two tools disagree at the margin because only 4 of the 80 queries differ (76 are unchanged): with so few informative pairs the evidence is weak, and the sensible decision is not to declare that fusion is worse, but to add queries before concluding. Finally the last column: the three comparisons are three tests, and every extra test increases the probability of finding something by chance. The Holm correction sorts the p-values and multiplies them by the number of tests still open; here it changes little because the effects are strong, but with four or five variants and modest differences it can make an apparent win vanish.

• Fix the evaluation set, the generation parameters and the seeds before looking at the results.
• Change one variable at a time: prompt and retriever together cannot be told apart.
• Repeat the generation several times if the temperature is above zero and aggregate, otherwise you are measuring noise.
• Decide beforehand the minimum difference you care about: a significant but irrelevant +0.01 does not justify a deploy.

These rules are the real content of the episode, and the code is only its support. The third is the one most often forgotten with an LLM in the middle: a judgment obtained at positive temperature is a sample, not a value, and a comparison between two single samples can reverse on repetition. The fourth prevents the opposite vice, declaring victory on a difference that is real but of no weight to the user.

Reranker, prompt and automatic optimization

The ablation of the reranker follows the same scheme: system without reranker against system with reranker, everything else equal, same per-query metric, same test. The cost to take into account is latency, because the cross-encoder scores each question-document pair with a pass of the model; how two-stage retrieval works, and where it stops, you find in this article. Late-interaction models, such as ColBERT, precompute the document representations offline, as the abstract of the original paper states: the space they occupy and the index they require are an operating cost to weigh, not a detail.

To keep track of prompts you have two paths. Prompt management tools, such as Langfuse, keep versions and labels (for example which version is in production) and let you link traces to the version used; a file versioned in git with a commit message reporting the change in metrics does the same job with fewer moving parts. Automatic optimization with DSPy searches for instructions and examples for you: MIPROv2, according to the project documentation, jointly optimizes instructions and few-shot examples by generating candidates and choosing the combination with Bayesian optimization. It needs a configured model, so the listing is a schema to adapt, and the metric you pass it becomes your constraint: if it is wrong, the optimizer optimizes it anyway.

# Schema to adapt, not executed: it needs a configured language model.
# Structure taken from the example in the MIPROv2 documentation (DSPy repository).
import dspy
from dspy.teleprompt import MIPROv2

dspy.configure(lm=dspy.LM("<provider>/<model>", api_key="<key>"))

def metric(example, prediction, trace=None):
    # your evaluation metric: returns a number (or a boolean) per example
    return float(example.expected_answer.lower() in prediction.answer.lower())

optimizer = MIPROv2(metric=metric, auto="light")      # light | medium | heavy
program = optimizer.compile(dspy.ChainOfThought("question, context -> answer"),
                            trainset=training_examples)
program.save("optimized_prompt.json")

Choosing in production with Thompson sampling

When you have several prompts already validated offline, a classic A/B test splits the traffic in half for a fixed time, even when one of the two is clearly losing. A multi-armed bandit moves the traffic toward the one that works. In Thompson sampling each prompt has a Beta distribution over its own success rate; at each request you draw one value for each and serve the one with the highest draw. The tutorial by Russo and co-authors explains the basics. The listing is a simulation: the true rates (0.60, 0.66, 0.72) are known only to whoever writes, and the "thumbs up" is generated with a fixed seed.

import numpy as np

# Simulation: three prompts with a true approval rate unknown to the system
TRUE = np.array([0.60, 0.66, 0.72])
rng = np.random.default_rng(3)
successes = np.ones(3)      # Beta(1, 1) prior for each prompt
failures = np.ones(3)
picks = np.zeros(3, dtype=int)

for _ in range(3000):
    samples = rng.beta(successes, failures)        # one draw per prompt
    k = int(np.argmax(samples))                    # serve the prompt with the highest draw
    outcome = rng.random() < TRUE[k]               # the observed "thumbs up"
    successes[k] += outcome
    failures[k] += 1 - outcome
    picks[k] += 1

post_mean = successes / (successes + failures)
for i, name in enumerate("ABC"):
    print(f"prompt {name}: served {picks[i]:4d} times, estimate {post_mean[i]:.3f} (true {TRUE[i]:.2f})")
print("mean approval obtained:", round(float((picks * TRUE).sum() / picks.sum()), 3),
      "against uniform:", round(float(TRUE.mean()), 3))

# Output:
# prompt A: served   66 times, estimate 0.574 (true 0.60)
# prompt B: served   12 times, estimate 0.357 (true 0.66)
# prompt C: served 2922 times, estimate 0.720 (true 0.72)
# mean approval obtained: 0.717 against uniform: 0.66

The best prompt receives 2922 requests out of 3000 and the approval obtained (0.717) exceeds that of a uniform split (0.660). But look at prompt B: served only 12 times, final estimate 0.357 against a true value of 0.66. A few unfavorable outcomes at the start pushed its distribution down and the algorithm stopped trying it: this is the other face of efficiency, and with a different seed it could happen to the best arm. That is why the bandit is used to choose among variants already judged acceptable, with real user feedback, and not to replace offline evaluation. The reward can be a thumbs up, a copy click or a judge score: choose the one that truly represents usefulness and is not easy to inflate.

Frequently asked questions

How many queries do I need for a reliable comparison?

There is no fixed number: it depends on the difference you want to detect and on the variability of the metric. The confidence interval is the practical guide: if it is as wide as the effect, you need more queries. Start from a few hundred, stratified by question type.

Why Wilcoxon and not the t-test?

Because retrieval and judgment metrics are bounded and often concentrated at the extremes, and the t-test assumes approximately normal differences. Wilcoxon requires only the symmetry of the differences under the null hypothesis.

Do I need the correction for multiple comparisons?

Yes when you compare more than two variants on the same set. The Holm correction is more powerful than Bonferroni for the same guarantee; all corrections reduce the risk of declaring a variant the winner by chance.

When is a bandit better than an A/B test?

When the variants are already validated offline, the traffic is sufficient and you want to limit the cost of exploration. To decide whether a change really holds up, a test designed in advance remains clearer.

What you can do now

You can set up a comparison between two variants with a single variable changed, compare them query by query, read difference, interval and correction for multiple tests together, recognize when a mean hides opposite segments and choose between A/B and bandit. In the next episode, monitoring production against the evaluation set, you use the same statistical tools on real traffic. For the infrastructure that runs these comparisons in batch, episode 7.