04 - Golden dataset: how to build eval set that predicts production quality

A convenient dataset gives a convenient estimate

Fifty questions picked on the fly, green scores, release. Two weeks later users complain about a third of the answers. The dataset was not wrong because of a calculation flaw: it had been written by someone who knows the system, and whoever knows the system phrases the questions the system understands. A useful golden dataset must satisfy three different conditions, and each is checked with a precise test: mirror the mix of questions that will really arrive, contain the cases in which the system fails often enough to measure them separately, and remain comparable over time, that is, not change secretly under your feet. This article builds a dataset in this order and uses, wherever possible, executed listings; the numbers come from a teaching model with declared assumptions and do not measure any real system.

Prerequisites: the concepts of confidence interval and sample size from episode 1 and the idea of a quality gate from episode 3. The next step, on the metrics of the retriever alone, will use the dataset built here.

How far off the estimate is, depending on how you choose the questions

Before any technique it is worth seeing the damage. The listing simulates a system with four types of question: easy, medium, hard and without an answer in the corpus. The traffic mix and the probability of a correct answer in each type are assumptions of the model (60/25/10/5 percent and 0.95/0.80/0.45/0.30), from which the real success rate is 0.830. Then it compares three ways of choosing forty questions, repeated five thousand times with a fixed seed. "Convenience" imitates whoever writes the questions by hand: it mostly picks the easy ones. "Random" draws them from the traffic. "Stratified" takes ten per type and weights each type with its share of the traffic.

# b2.py - teaching model: how far off is the success-rate estimate with three ways of choosing 40 questions
import numpy as np

WEIGHTS = np.array([0.60, 0.25, 0.10, 0.05])     # traffic mix: easy, medium, hard, unanswerable
P_OK = np.array([0.95, 0.80, 0.45, 0.30])        # ASSUMPTION: probability that the system answers well in each stratum
TRUE = float(WEIGHTS @ P_OK)                       # real success rate in production
N, REPETITIONS = 40, 5000
rng = np.random.default_rng(2026)

def convenience() -> float:
    """Whoever writes the questions by hand picks their own: 85% easy, 15% medium, never hard."""
    strata = rng.choice(4, size=N, p=[0.85, 0.15, 0.0, 0.0])
    return rng.binomial(1, P_OK[strata]).mean()

def random_sample() -> float:
    """Random sample from the traffic, simple mean."""
    strata = rng.choice(4, size=N, p=WEIGHTS)
    return rng.binomial(1, P_OK[strata]).mean()

def stratified() -> float:
    """10 questions per stratum, then a mean weighted with the traffic mix."""
    per_stratum = rng.binomial(10, P_OK) / 10
    return float(WEIGHTS @ per_stratum)

print(f"real success rate (model assumption): {TRUE:.3f}")
for name, f in (("convenience", convenience), ("random", random_sample), ("stratified", stratified)):
    estimates = np.array([f() for _ in range(REPETITIONS)])
    print(f"{name:12s} mean of the estimates={estimates.mean():.3f}  bias={estimates.mean() - TRUE:+.3f}  standard deviation={estimates.std():.3f}")

# Output:
# real success rate (model assumption): 0.830
# convenience  mean of the estimates=0.928  bias=+0.098  standard deviation=0.041
# random       mean of the estimates=0.831  bias=+0.001  standard deviation=0.060
# stratified   mean of the estimates=0.830  bias=-0.000  standard deviation=0.055

Convenience is not just imprecise, it is biased: it estimates 0.928 against a real value of 0.830, about ten points too high, and with a low dispersion that makes it look reliable. It is the mechanism of "all the metrics were green". The random sample has no bias, but it has the highest dispersion (0.060 standard deviation). The stratified one has a dispersion only a little lower (0.055); in this model the gain on the overall number is modest, and it is not the main reason to stratify. The reason is that with ten questions per type you see where the system fails: the overall number hides that the hard questions stand at 0.45, and that is the only datum you can act on. Weighting with the traffic mix requires knowing it, so you need a log of real questions, even a small one.

Stratified sampling with explicit quotas

Generating synthetic questions with Ragas, described in episode 2, can provide a first draft of the strata, but with the construction flaw explained there: every synthetic question already has an answer in the corpus. The sampler does three things that a random.sample does not. It takes a fixed quota from each stratum, it sorts the questions by identifier before drawing, so that the result depends only on the seed and not on the order of the file, and it fails with a clear message if a stratum has fewer questions than the requested quota: silently filling in with other questions would falsify the stratification without anyone noticing. The invented log has 400 questions in the proportions of the previous example.

# b1.py - stratified sampling with explicit quotas and a fixed seed
import random
from collections import Counter

STRATA = ("easy", "medium", "hard", "unanswerable")

def invented_log(n: int = 400, seed: int = 1) -> list:
    """A log of example questions: id and stratum, in the proportions 60/25/10/5."""
    rng = random.Random(seed)
    weights = (0.60, 0.25, 0.10, 0.05)
    return [{"id": f"d{i:03d}", "stratum": rng.choices(STRATA, weights)[0]} for i in range(n)]

def sample(questions: list, quotas: dict, seed: int = 42) -> list:
    """Draws quotas[stratum] questions from each stratum. If a stratum has fewer, it fails: it does not fill in silently."""
    rng = random.Random(seed)
    chosen = []
    for stratum, n in quotas.items():
        pool = sorted((d for d in questions if d["stratum"] == stratum), key=lambda d: d["id"])
        if len(pool) < n:
            raise ValueError(f"stratum '{stratum}': {n} questions needed, only {len(pool)} available")
        chosen += rng.sample(pool, n)
    return sorted(chosen, key=lambda d: d["id"])

log = invented_log()
print("log:", dict(sorted(Counter(d["stratum"] for d in log).items())))

at_random = random.Random(7).sample(log, 40)
print("40 at random:   ", dict(sorted(Counter(d["stratum"] for d in at_random).items())))

stratified = sample(log, {"easy": 10, "medium": 10, "hard": 10, "unanswerable": 10})
print("10 per stratum: ", dict(sorted(Counter(d["stratum"] for d in stratified).items())))

try:
    sample(log, {"unanswerable": 500})
except ValueError as error:
    print("impossible quota ->", error)

# Output:
# log: {'easy': 250, 'hard': 40, 'medium': 89, 'unanswerable': 21}
# 40 at random:    {'easy': 25, 'hard': 5, 'medium': 6, 'unanswerable': 4}
# 10 per stratum:  {'easy': 10, 'hard': 10, 'medium': 10, 'unanswerable': 10}
# impossible quota -> stratum 'unanswerable': 500 questions needed, only 21 available

The proportions of the log are 250 easy, 89 medium, 40 hard and 21 unanswerable. A random sample of forty takes only four of the last type and five of the hard ones: too few to say anything about the stratum, and not out of bad luck but by construction. The fixed quota of ten per stratum gives each one an estimate with its own interval, as wide as that of a sample of ten. A sample of ten questions has a very wide interval (with the formula from episode 1, for a proportion of 0.8 the 95% interval goes from about 0.49 to 0.94), so ten per stratum is the starting point, not the goal. The last line shows the error it prevents: asking for 500 unanswerable questions from a log that has 21 raises an exception. The strata come from your log or from interviews with users, not from a standard list: the example uses four because they are enough to show the mechanism.

Annotating: the schema, the validator, the agreement between people

A question with its right answer is a datum that someone decided. The schema must keep what is needed to diagnose an error months later: the stratum, whether the question is answerable, the documents that contain the answer, the reference answer, who annotated and when. The answerable field is what separates the questions the system must answer from those it must refuse, and without it a correct refusal is counted as an error. The corpus fingerprint ties the dataset to the version of the documents on which it was annotated: if a document changes, the fingerprint changes and the references must be re-verified.

# b3.py - one golden dataset record, its validator and the corpus fingerprint
import hashlib, json
from sklearn.metrics import cohen_kappa_score

CORPUS = {
    "pricing": "The Pro plan costs 49 euros a month and includes 10 users.",
    "refund": "A refund is possible within 14 days of purchase, by contacting support.",
    "api": "The API accepts at most 100 requests per minute for each key.",
}

def corpus_fingerprint(corpus: dict) -> str:
    """Stable hash of the corpus: it changes if even a comma of a document changes."""
    block = json.dumps(corpus, sort_keys=True, ensure_ascii=False).encode("utf-8")
    return "sha256:" + hashlib.sha256(block).hexdigest()[:16]

FIELDS = {"id": str, "question": str, "stratum": str, "answerable": bool,
          "expected_doc_ids": list, "reference_answer": str, "annotator": str, "date": str}

def validate(record: dict, corpus: dict, seen: set) -> list:
    errors = []
    for field, kind in FIELDS.items():
        if field not in record:
            errors.append(f"missing '{field}'")
        elif not isinstance(record[field], kind):
            errors.append(f"'{field}' must be {kind.__name__}")
    if record.get("id") in seen:
        errors.append(f"duplicate id '{record['id']}'")
    seen.add(record.get("id"))
    if record.get("answerable") and not record.get("expected_doc_ids"):
        errors.append("answerable question without expected documents")
    if not record.get("answerable") and record.get("expected_doc_ids"):
        errors.append("unanswerable question with expected documents")
    for d in record.get("expected_doc_ids", []):
        if d not in corpus:
            errors.append(f"expected document '{d}' does not exist in the corpus")
    return errors

DATASET = {
    "schema_version": 1,
    "corpus": corpus_fingerprint(CORPUS),
    "records": [
        {"id": "q001", "question": "How many users does the Pro plan include?", "stratum": "easy", "answerable": True,
         "expected_doc_ids": ["pricing"], "reference_answer": "It includes 10 users.",
         "annotator": "A", "date": "2026-11-12"},
        {"id": "q002", "question": "Does the Pro plan include phone support?", "stratum": "unanswerable", "answerable": False,
         "expected_doc_ids": [], "reference_answer": "The corpus does not say: the system must refuse.",
         "annotator": "A", "date": "2026-11-12"},
        {"id": "q002", "question": "How does the refund work?", "stratum": "easy", "answerable": True,
         "expected_doc_ids": ["refunds"], "reference_answer": "Within 14 days.",
         "annotator": "B", "date": "2026-11-12"},
    ],
}

seen = set()
for r in DATASET["records"]:
    errors = validate(r, CORPUS, seen)
    print(r["id"], "OK" if not errors else errors)
print("corpus fingerprint:", DATASET["corpus"])
CORPUS["api"] = CORPUS["api"].replace("100", "200")
print("after a change:", corpus_fingerprint(CORPUS))

# two people label the same 10 questions as "correct answer" (1) or not (0): invented labels
a = [1, 1, 0, 1, 1, 0, 1, 0, 1, 1]
b = [1, 1, 0, 1, 0, 0, 1, 0, 1, 1]
agreement = sum(x == y for x, y in zip(a, b)) / len(a)
print(f"raw agreement={agreement:.0%}  Cohen's kappa={cohen_kappa_score(a, b):.2f}")

# Output:
# q001 OK
# q002 OK
# q002 ["duplicate id 'q002'", "expected document 'refunds' does not exist in the corpus"]
# corpus fingerprint: sha256:a26e765c49c3a357
# after a change: sha256:1b8a33f122b71db1
# raw agreement=90%  Cohen's kappa=0.78

The validator is where the schema stops being a convention. It checks that every field is there and has the right type, that identifiers are unique, that an answerable question has at least one expected document and that an unanswerable one has none, and that the expected documents really exist in the corpus. In the example the third record has a duplicate identifier and cites a document ("refunds") that in the corpus is called "refund": two errors that produce a silently wrong dataset, because the retriever will never find it and the score will collapse because of the annotation. The last lines show the fingerprint before and after changing a number in a document: they are different, and the difference is what tells you to redo the check.

The agreement between people remains. If you have no domain experts, have the same subset annotated by two people and measure the concordance with Cohen's kappa, which corrects the raw agreement for the agreement expected by chance: in the example, 90% raw agreement becomes a kappa of 0.78. The scikit-learn documentation defines it as (p_o − p_e) / (1 − p_e), with p_o the observed agreement and p_e the expected one. A low kappa does not say who is right, it says that the question or the criterion is ambiguous; discuss it before trusting the score that derives from it. The labels in the example are invented, and the kappa on ten questions is itself very uncertain.

The hard cases: perturbing the questions

Strata describe the difficulty of the question; perturbations describe its form. The same question can arrive with a typo, in capitals, preceded by courtesy phrases, truncated or rephrased with different words. The listing takes ten questions, each with its right document in a corpus of ten, and submits them to TF-IDF retrieval in six forms. The paraphrase is written by hand for each question, because an automatic paraphrase would introduce a second model to validate.

# b4.py - the same questions, perturbed: how well lexical retrieval holds up
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

DOCS = {
    "pricing":      "The Pro plan costs 49 euros a month and includes 10 users.",
    "refund":       "A refund is possible within 14 days of purchase, by contacting support.",
    "export":       "Data export produces a CSV file from the Settings page.",
    "sso":          "SSO login is available only on the Enterprise plan through SAML.",
    "backup":       "Automatic backups run every night and are kept for 30 days.",
    "api":          "The API accepts at most 100 requests per minute for each key.",
    "invoices":     "Invoices can be downloaded as PDF from the Billing section of the account.",
    "password":     "To reset your password use the link sent to your registration email.",
    "integrations": "Integrations with Slack and Teams are enabled from the Integrations panel.",
    "limits":       "Each project accepts up to 500 documents uploaded in PDF format.",
}
# (canonical question, hand-written paraphrase, right document)
CASES = [
    ("how much does the Pro plan cost per month", "what monthly price does Pro have", "pricing"),
    ("how do I download my data as CSV", "I want a CSV file with the data", "export"),
    ("is SAML required to log in", "SSO login is on the Enterprise plan", "sso"),
    ("how often are backups made", "automatic backups are nightly", "backup"),
    ("how many requests per minute does the API accept", "limit of requests per minute for each key", "api"),
    ("where do I download invoices as PDF", "download PDF invoices billing", "invoices"),
    ("I forgot my password", "reset the forgotten password", "password"),
    ("how do I connect Slack", "enable the integration with Slack", "integrations"),
    ("how many documents can I upload per project", "maximum number of documents per project", "limits"),
    ("can I get my money back", "how does the refund work", "refund"),
]

def typo(q: str) -> str:
    """Swaps two adjacent letters in the longest word (the typical typo)."""
    words = q.split()
    i = max(range(len(words)), key=lambda k: (len(words[k]), -k))
    w = words[i]
    m = len(w) // 2
    words[i] = w[: m - 1] + w[m] + w[m - 1] + w[m + 1:]
    return " ".join(words)

PERTURBATIONS = {
    "original":     lambda c, p: c,
    "paraphrase":   lambda c, p: p,
    "typo":         lambda c, p: typo(c),
    "uppercase":    lambda c, p: c.upper(),
    "with courtesy": lambda c, p: "hi sorry, a quick question: " + c + ", thanks",
    "truncated":    lambda c, p: " ".join(c.split()[:3]),
}

ids = list(DOCS)
vec = TfidfVectorizer(sublinear_tf=True)
M = vec.fit_transform(DOCS.values())
for name, f in PERTURBATIONS.items():
    hit = 0
    for canonical, paraphrase, right in CASES:
        scores = (M @ vec.transform([f(canonical, paraphrase)]).T).toarray().ravel()
        hit += right in [ids[i] for i in np.argsort(-scores, kind="stable") if scores[i] > 0][:3]
    print(f"{name:14s} hit@3={hit}/{len(CASES)}")

# Output:
# original       hit@3=9/10
# paraphrase     hit@3=10/10
# typo           hit@3=8/10
# uppercase      hit@3=9/10
# with courtesy  hit@3=9/10
# truncated      hit@3=4/10

The results contradict the common intuition: lexical retrieval holds up well to capitals and courtesies (nine questions out of ten, as for the original text), because many words of the question stay intact, and it loses one question to the typo (eight out of ten). It drops sharply on the version truncated to three words (four out of ten): in English the questions begin with function words ("how do I", "how many"), so cutting to three words discards the content words. The question already missed in its original form, "can I get my money back", is a vocabulary problem, not a form problem: the paraphrase retrieves it (ten out of ten) only because I wrote it with the word "refund", the same as in the document, so that 10/10 demonstrates the mechanism and not robustness.

You draw a working rule from it: perturbations serve to measure which forms your retriever cannot stand, and the answer changes with the retriever; an embedding model behaves differently and must be measured again. Always test each perturbation against the original and report the difference, not the absolute value.

Versioning the dataset

The dataset is a file, and a file is versioned. As long as it is a JSON of a few hundred records, the simplest solution is to put it in Git together with the code, with the schema version and the corpus fingerprint in the file and a Git tag at each release of the dataset; every reported score must cite the version with which it was computed. When the dataset or the corpus becomes large, DVC offers an alternative: its documentation explains that it is not a version control system in itself, but manipulates .dvc files that define the versions of the data and that Git can version, while the data lives elsewhere. There is no need for DVC before it is needed.

Does the dataset predict production? A check with few points

The only way to know whether the golden dataset works is to compare it with what happens in production. You have a series of versions of the system; for each you have the score on the dataset and a production signal that you choose (the share of answers judged useful by users, for example). If the two move together, the dataset is looking at the right thing. The listing computes the Pearson correlation with its confidence interval, the Spearman one, which looks only at the order of the versions, and shows how the interval would narrow with more versions. The data are invented: they serve to show the method.

# b5.py - does the golden dataset predict production? The correlation across versions, with its interval
import math
from scipy.stats import pearsonr, spearmanr

# INVENTED DATA to illustrate the method: 8 versions of the system.
# score on the golden dataset and share of answers judged useful in production (from a signal of your choice)
eval_score = [0.62, 0.68, 0.71, 0.74, 0.77, 0.80, 0.83, 0.86]
useful_in_production = [0.55, 0.66, 0.60, 0.72, 0.70, 0.78, 0.74, 0.81]

r = pearsonr(eval_score, useful_in_production)
lo, hi = r.confidence_interval(confidence_level=0.95)
rho = spearmanr(eval_score, useful_in_production)
print(f"versions={len(eval_score)}")
print(f"Pearson r={r.statistic:.2f}  95% interval=[{lo:.2f}, {hi:.2f}]")
print(f"Spearman rho={rho.statistic:.2f}  (does it agree on the order of the versions?)")

# if you had 24 with the same correlation, the interval would narrow like this (Fisher z):
z = math.atanh(r.statistic)
for n in (8, 24, 60):
    s = 1 / math.sqrt(n - 3)
    print(f"n={n:3d}: [{math.tanh(z - 1.96 * s):.2f}, {math.tanh(z + 1.96 * s):.2f}]")

# Output:
# versions=8
# Pearson r=0.92  95% interval=[0.61, 0.99]
# Spearman rho=0.93  (does it agree on the order of the versions?)
# n=  8: [0.61, 0.99]
# n= 24: [0.82, 0.97]
# n= 60: [0.87, 0.95]

With eight versions the correlation of 0.92 has an interval from 0.61 to 0.99: the direction is clear, the strength is not. Spearman gives 0.93, that is, the order of the versions is almost the same in the two series. With twenty-four versions and the same correlation the interval would go from 0.82 to 0.97, with sixty from 0.87 to 0.95. You will never have sixty versions: but you can use subsets of the traffic or time windows as additional points, remembering that they are not independent. Correlation does not demonstrate a cause, and a high value with the wrong production signal (for example the length of the answer) is worth nothing: choose the signal before looking at the results.

What you can do now

You can show, with a simulation you can redo, why a hand-written dataset overestimates quality, and you can build a stratified one with explicit quotas, a fixed seed and a clear error when a stratum is not enough. You can write a schema that distinguishes the questions to answer from those to refuse, validate it with code, tie it to a corpus fingerprint and measure the agreement between annotators. You can perturb the questions to see which forms your retriever cannot take, and verify with a confidence interval whether the score on the dataset follows production. For the part that concerns continuous observation after release you can continue with Evaluating RAG: metrics, benchmarking and production monitoring.

The same series also includes evaluation automation.