09 - Monitoring production vs eval set: drift detection, retraining trigger

From laboratory evaluation to production monitoring

On your evaluation set the RAG system does well. Two weeks after release someone writes that "the answers have become strange". You pull a sample of real requests, evaluate them, and the quality is lower than in the laboratory. No bug: the evaluation set was a snapshot of the domain at the moment you built it, while users ask questions you hadn't foreseen, the corpus changes and expectations shift. The gap between what you measure and what happens is called in this episode the evaluation-production gap, and it is the problem that closes the series.

You build four pieces with numbers you can redo: stable sampling of the traffic, a statistical check on the shift of queries, an intervention rule that requires two agreeing signals, and a comparison between the current and a candidate version. To measure the metrics themselves (faithfulness, relevance) I refer you to episode 6 and to how to run them in batch in episode 7; for the general version of the topic in the AI Engineering series, this article on RAG in production. The examples use invented queries and no model: it is the mechanism that matters, not the figures.

Three kinds of shift, three different responses

Calling everything "drift" hides the fact that causes and remedies differ. The table separates the three cases you meet most often.

This episode measures the first case with a statistical test, because it is the only one you can see without looking at the answers. The other two emerge from the quality metrics and from inspecting the corpus: a useful indicator for the second is the share of queries whose best chunk has a low score, which flags gaps in the documents.

Sampling without slowing down and without distorting

You don't evaluate every request: you sample a fraction. The simplest way, picking at random with a generator, has two defects: it is not reproducible and two replicas of the service decide differently. Better a deterministic decision on the request identifier: you compute a hash, reduce it to a fixed scale and compare with the share. The same identifier always gives the same answer, and raising the share from 5% to 10% includes all the requests already sampled.

import hashlib


def should_sample(request_id, share):
    """Deterministic decision: the same id gets the same answer, on every replica."""
    h = int.from_bytes(hashlib.sha256(request_id.encode()).digest()[:8], "big")
    return (h % 10_000) < share * 10_000


ids = [f"req-{i}" for i in range(20_000)]
chosen = [i for i in ids if should_sample(i, 0.05)]
print(f"sampled {len(chosen)} requests out of {len(ids)} ({len(chosen) / len(ids):.2%}), requested share 5%")
print("same decision on the second call:", all(should_sample(i, 0.05) for i in chosen))
print("with a 10% share I also include all the previous ones:", all(should_sample(i, 0.10) for i in chosen))

# Output:
# sampled 961 requests out of 20000 (4.81%), requested share 5%
# same decision on the second call: True
# with a 10% share I also include all the previous ones: True

Out of twenty thousand identifiers, 961 were chosen, that is 4.81% against a requested 5%: the gap is normal for pseudorandom sampling and is not an error. The two check lines show the properties you need: stability and nesting. The cost on the request path is a hash and a comparison; the actual writing and the evaluation must be done outside the path, in batch. Two warnings that no listing solves: the sampled requests may contain personal data and must be treated accordingly (minimize, anonymize, set a retention period), and uniform sampling under-represents rare topics, so for the important segments a higher share is advisable.

Measuring the shift of queries

Comparing two sets of high-dimensional vectors with one test per dimension produces hundreds of comparisons. You reduce it to a single number per query: the cosine similarity between the query and the centroid of the evaluation set. What remains is a one-dimensional distribution, comparable with the two-sample Kolmogorov-Smirnov test (SciPy's ks_2samp), which measures the maximum distance between the two cumulative distribution functions, and with the Population Stability Index (PSI), which sums the gap between the shares of each interval weighted by the logarithm of their ratio.

In the listing below the queries are invented. The evaluation set has 200 queries on five technical topics, with different weights as in real traffic; week 1 has the same distribution; week 2 has 40% of queries on topics never seen (invoices, SPID, IBAN, VAT rates). The centroid is computed with the first half of the set and the reference is the second half: if you computed the reference scores on the same queries that built the centroid, the reference would come out closer to the center than new traffic ever will be, and you would see drift where there is none. The PSI intervals are defined by the quantiles of the reference.

import numpy as np
from scipy.stats import ks_2samp
from sklearn.feature_extraction.text import TfidfVectorizer

OBJECTS = ["cache", "database", "log", "certificate", "backup"]
ACTIONS = ["configure", "restart", "monitor", "update", "delete", "import", "export", "protect"]
NEW = ["issue electronic invoice", "activate spid", "verify iban", "calculate vat rate"]

rng = np.random.default_rng(1)
WEIGHTS = [0.40, 0.25, 0.15, 0.10, 0.10]       # in real traffic some topics dominate


def traffic(n, new_share=0.0):
    out = []
    for _ in range(n):
        if rng.random() < new_share:
            out.append(NEW[rng.integers(len(NEW))])
        else:
            out.append(f"{ACTIONS[rng.integers(8)]} {rng.choice(OBJECTS, p=WEIGHTS)}")
    return out


evaluation = traffic(200)                      # evaluation set, built once
week1 = traffic(200)                           # same distribution
week2 = traffic(200, new_share=0.4)            # 40% of the queries are about new topics

vec = TfidfVectorizer().fit(evaluation)
V = vec.transform(evaluation[:100])            # the centroid uses only the first half
centroid = np.asarray(V.mean(axis=0)).ravel()
centroid /= np.linalg.norm(centroid)


def scores(queries):
    """Cosine between each query and the centroid of the evaluation set."""
    return np.asarray(vec.transform(queries) @ centroid).ravel()


def psi(reference, observed, bins=5):
    """Population Stability Index on bins defined by the quantiles of the reference."""
    edges = np.unique(np.quantile(reference, np.linspace(0, 1, bins + 1)))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(reference, edges)[0] / len(reference)
    o = np.histogram(observed, edges)[0] / len(observed)
    e, o = np.clip(e, 1e-4, None), np.clip(o, 1e-4, None)
    return float(((o - e) * np.log(o / e)).sum())


ref = scores(evaluation[100:])                  # queries held out of the centroid
s1, s2 = scores(week1), scores(week2)
print("distinct score values in the reference:", len(set(np.round(ref, 6))), "out of", len(ref))
for name, s in [("week 1", s1), ("week 2", s2)]:
    ks = ks_2samp(ref, s)
    print(f"{name}: mean cosine={s.mean():.2f}  KS={ks.statistic:.2f} (p={ks.pvalue:.2g})  PSI={psi(ref, s):.2f}")
print("week 2 queries with cosine 0:", int((s2 == 0).sum()), "out of", len(s2))

# Output:
# distinct score values in the reference: 33 out of 100
# week 1: mean cosine=0.39  KS=0.20 (p=0.0069)  PSI=0.21
# week 2: mean cosine=0.24  KS=0.45 (p=2.6e-12)  PSI=0.73
# week 2 queries with cosine 0: 76 out of 200

Week 2 stands out clearly: KS 0.45, PSI 0.73 and 76 queries out of 200 with cosine zero, that is, with no words in common with the evaluation set. Week 1, which comes from the same generator, nevertheless gives KS 0.20 with p = 0.0069: the test would flag it. This is the limit to know. KS presupposes continuous distributions, while here the scores take only 33 distinct values out of 100: the many equal values make the p-value optimistic. The practical lesson is to look at the statistic and not only at the p-value (0.20 against 0.45), and to set the threshold on what you observe in the quiet periods of your traffic. With very large samples the opposite holds: the test becomes sensitive even to negligible differences, and once again the size of the gap is what counts. The PSI thresholds you read around are conventions, not norms.

When to intervene: two signals must agree

The shift of queries says that the world has changed, not that the system is getting worse. An alarm on every shift produces noise, and after three false alarms nobody looks at the fourth. The rule in the listing intervenes only if two conditions are true together: the shift exceeds a relevance threshold (high KS statistic and low p-value) and the quality metric, measured on the sample, stays below the SLA for two consecutive checks. In all other cases the state is "observe" or "no action".

def action(history):
    """history: list of checks with 'ks', 'p', 'metric'. Acts only if the two signals agree."""
    SLA, KS_MAX = 0.70, 0.40
    last = history[-2:]
    drift = history[-1]["ks"] > KS_MAX and history[-1]["p"] < 0.01
    below_sla = len(last) == 2 and all(c["metric"] < SLA for c in last)
    if drift and below_sla:
        return "ACT (update eval set and index)"
    if drift or history[-1]["metric"] < SLA:
        return "OBSERVE"
    return "NONE"


checks = [
    {"ks": 0.12, "p": 0.40, "metric": 0.78},
    {"ks": 0.45, "p": 0.001, "metric": 0.74},    # drift, but quality holds
    {"ks": 0.47, "p": 0.001, "metric": 0.69},    # first measurement below the SLA
    {"ks": 0.50, "p": 0.001, "metric": 0.66},    # second in a row: the two signals agree
]
for n in range(1, len(checks) + 1):
    print(f"check {n}:", action(checks[:n]))

# Output:
# check 1: NONE
# check 2: OBSERVE
# check 3: OBSERVE
# check 4: ACT (update eval set and index)

At the second check there is already drift (KS 0.45) but quality holds at 0.74, so you only observe. At the third quality falls to 0.69, the first measurement below the SLA: you still observe. At the fourth, 0.66 is the second consecutive measurement below the SLA and the drift is confirmed: the action fires. The SLA of 0.70 and the KS threshold of 0.40 are example values. To choose them without arbitrariness, repeat the measurement on the same set several times to estimate the natural variability of the metric (an LLM is not deterministic) and set the threshold beyond that fluctuation. The action is not "retrain" by default: in RAG it is almost always updating the evaluation set, rebuilding the index or fixing the documents, and the model is touched last.

Trying a new version on part of the traffic

When the change passes the offline tests, the risk remains of what the evaluation set does not cover. Route a small share of the traffic to the candidate version (here 10%), collect the scores for a fixed period and compare the distributions. The Mann-Whitney U test (scipy.stats.mannwhitneyu) is non-parametric: its null hypothesis is that the two distributions are equal, and it compares not the means but the ranks, so it holds up better against skewed scores and a few extreme values. Next to the p-value it is worth looking at the probability that a query of the candidate beats one of the current version: U divided by the product of the two sample sizes.

import numpy as np
from scipy.stats import mannwhitneyu

rng = np.random.default_rng(5)
# Per-request scores (simulated): version B is slightly better in the low tail
a = np.clip(rng.beta(7, 3, size=900), 0, 1)       # 90% of the traffic, current version
b = np.clip(rng.beta(7.6, 3, size=100), 0, 1)     # 10% of the traffic, candidate

r = mannwhitneyu(b, a, alternative="two-sided")
print(f"n_A={len(a)} n_B={len(b)}  mean A={a.mean():.3f}  mean B={b.mean():.3f}")
print(f"U={r.statistic:.0f}  p={r.pvalue:.3f}  probability that a B query beats an A query={r.statistic / (len(a) * len(b)):.3f}")
print("share of requests with score < 0.4: A", round(float((a < 0.4).mean()), 3), " B", round(float((b < 0.4).mean()), 3))

# Output:
# n_A=900 n_B=100  mean A=0.702  mean B=0.710
# U=46459  p=0.595  probability that a B query beats an A query=0.516
# share of requests with score < 0.4: A 0.028  B 0.02

Here the candidate has a mean of 0.710 against 0.702, p = 0.595 and probability of superiority 0.516: no demonstrable difference with 100 requests. It is the most frequent result, and it says something important: with a small share and little time you cannot tell small improvements apart, and the test protects you from promoting on faith. The most useful comparison for safety looks at the tail, the share of requests with a low score (2.8% against 2.0%): a serious regression shows up there before it does in the mean. The scores are simulated with fixed seeds; on your production the figures will be different. Always provide an immediate return to the previous version, with a single switch.

What to put on the dashboard

A dashboard serves to decide whether to look elsewhere, not to show everything. For a RAG a few series over time are enough: the quality metrics of the sample with the SLA band, the query shift statistic, the share of queries without good retrieval results and the sampled volume. Add every series that has already anticipated a real problem, not those that "might be useful". Metrics are exported to the monitoring system you already use; the choice of tool matters less than three habits: written and versioned thresholds, release annotations on the chart, and an owner who receives the alarm.

Frequently asked questions

What is the difference between data drift and concept drift in a RAG?

Data drift is a change in the distribution of incoming queries, and you see it by comparing the queries with those of the evaluation set. Concept drift is a change in the correct answer (prices, rules, versions): it shows up as obsolete answers even when retrieval finds the document you expect.

How much should I sample?

Enough to have at least a few dozen samples for each check, at a frequency that lets you react. With little traffic raise the share or aggregate by week; below a few dozen samples statistical tests have little power and it is better to wait.

Is the Kolmogorov-Smirnov test always suitable?

It is designed for continuous distributions, so with many repeated values the p-value becomes approximate, as in the example. Use it together with the size of the gap and with thresholds calibrated on quiet traffic.

When is it better to stop and observe?

When the two signals don't agree: a shift in queries with stable quality is useful information for widening the evaluation set, not a reason to change the system.

What you can do now

You can tell the three kinds of shift apart and link each to a remedy, sample the traffic reproducibly, measure the shift of queries with KS and PSI knowing the limits of the test, decide an intervention only when shift and quality agree, and compare a candidate version with Mann-Whitney on the distributions, looking at the tail too. With this the series closes: from the question why "it works" is not a metric to a measurement cycle that continues after release.

The same series also includes optimization driven by evaluation and re-ranking with cross-encoders.