98 - Evaluating a Generative System: the Golden Set, the Automatic Judge and Its Biases

You changed the prompt of a system that answers questions about your documents, and by eye the answers look better. Before you tell anyone, you should be able to prove it, and this is where three problems arise that almost nobody separates: how many questions you need to see a difference of five points, who decides that a reworded answer is right, and how far you can trust an automatic judge that decides it for you. A judge that is wrong in the same direction every time does not just err: it hands you a wrong ranking with confidence.

The lesson on RAG built a system that retrieves and cites; this one asks how you can say it has improved. I take the corpus and the 18 questions of that lesson and widen them into a golden set of 36 cases, I work out which differences I can read, I compare three automatic metrics with my own labels, I measure the agreement of different judges with Cohen's kappa, and I try to find the three biases that the literature attributes to automatic judges. I close with the regressions between two versions of the same system, which is the everyday use of everything else.

How the examples are written, and what they can prove. No model runs in this lesson and no service is called. The three systems to evaluate are a scripted extractor (modello_finto, the same one as in the RAG lesson), a version of it with a higher abstention threshold, and a table of answers I wrote myself; the judges are programs that err with probabilities I fix, except one, a rule-based judge, which is a real program. The seed is fixed and the whole program, made of the blocks of this page in the order in which they appear, was run twice with identical output. Versions: Python 3.14.4, NumPy 2.3.5, SciPy 1.16.3.

A word on the data and on the names. The corpus, the questions and the answers are in English because they come from the docstrings of the Python standard library; what you will find in Italian, as in the Italian edition of this lesson, are the identifiers of the code (modello_finto, casi, giudici) and the names of some simulated judges, which feed the seed of every random decision. I left them untouched on purpose: translating them would change the seeds, and the figures you read here would no longer be the ones of the Italian lesson.

This changes the nature of what I demonstrate. With simulated judges I can show you the method: how to size a golden set, how to compute an interval, how to recognise a lazy judge from its kappa, how to find out whether a bias is there or not. I cannot tell you how large the position bias of a real model is, because the bias I find is the one I wrote into the parameters. Where a result holds only with a real model, I attribute it to the papers that measured it, and I flag it to you; where a model is needed, the code is a template to adapt and it has not been run.

What is real and what is fake in this lesson

• Corpus and questions: real. The docstrings of the standard library, extracted with inspect, and the 18 questions of the RAG lesson, each with the literal anchor that contains the answer.
• Systems A and A2: real, deterministic programs. They take the sentence of the context that shares the most words with the question and abstain below a threshold (2 for A, 3 for A2). They are not models.
• System C: a table of 20 answers written and judged by me. It exists to produce what an extractor does not produce: right answers in words different from the reference, and a few answers that are plausible and wrong.
• Judges G1, G2, G3: simulated, with error rates fixed in the code. The rule-based judge (token F1 with a threshold) is instead a real program.
• Biases: my own parameters. Measuring them means checking that the protocol finds them again.

Read the list as a map of trust. Everything about counts, intervals, tests and lexical metrics is real computation on real data, and you can repeat it identically. Everything about how much a judge errs is a hypothesis that I wrote: it is there to try the measuring instruments on a case where the right answer is known, which is also how the real ones are tested.

The golden set: 36 cases built from the corpus

A golden set is a set of cases in which you decide the right answer: a question, a context and the reference that contains the answer. You cannot buy it, and you cannot generate it with a model, because if you generate it you measure the agreement between two models and call it accuracy. The corpus is the one from the RAG lesson: eighteen modules of the standard library, each reduced to its docstrings.

import collections, hashlib, importlib, inspect, math, re, statistics, time
import numpy as np

MODULI = ["statistics", "bisect", "heapq", "itertools", "functools", "difflib",
          "textwrap", "tempfile", "uuid", "base64", "secrets", "gzip", "math",
          "glob", "zlib", "operator", "fractions", "fnmatch"]

def normalizza(s):
    return " ".join(s.split())

def costruisci_corpus():
    documenti, visti = [], set()
    for nome in MODULI:
        m = importlib.import_module(nome)
        parti = [f"MODULE {nome}"]
        if inspect.getdoc(m):
            parti.append(inspect.getdoc(m).strip())
        for attr, obj in sorted(vars(m).items()):
            if attr.startswith("_"):
                continue
            if not (inspect.isfunction(obj) or inspect.isclass(obj) or inspect.isbuiltin(obj)):
                continue
            doc = inspect.getdoc(obj)
            if not doc or len(doc.split()) < 20:
                continue
            casa = obj.__module__.lstrip("_")               # the module that defines it
            if casa != nome and casa in MODULI:             # re-exported from a module that
                continue                                    # is in the corpus: its owner keeps it
            impronta = (obj.__module__, obj.__qualname__)   # then each object only once
            if impronta in visti:
                continue
            visti.add(impronta)
            parti.append(f"## {nome}.{attr}\n\n{doc.strip()}")
        documenti.append({"id": nome, "testo": "\n\n".join(parti), "stato": "corrente"})
    return documenti

The function keeps each object once, in the module that defines it, and discards docstrings under twenty words: a line such as "Return the sum" cannot contain an answer worth checking. The filter on the module of origin prevents functools and statistics from fighting over the same re-exported function.

# (question, module, anchor): the anchor is the literal string that contains the answer
GOLDEN = [
    ("What is the default cutoff of get_close_matches?", "difflib",
     "Optional arg cutoff (default 0.6)"),
    ("Which position does bisect_left return when x is already in the list?", "bisect",
     "insert just before the leftmost x already there"),
    ("Where does insort_right put an item that is already present?", "bisect",
     "insert it to the right of the rightmost x"),
    ("How does heappushpop compare with heappush followed by heappop?", "heapq",
     "runs more efficiently than heappush() followed by a separate call to heappop()"),
    ("What does fmean do when the dataset is empty?", "statistics",
     "If the input dataset is empty, it raises a StatisticsError."),
    ("What is the average speed from harmonic_mean for 40 km/hr and 60 km/hr over equal distances?", "statistics",
     "harmonic_mean([40, 60]) 48.0"),
    ("What does median_low return with an even number of data points?", "statistics",
     "the smaller of the two middle values is returned"),
    ("How does textwrap shorten treat whitespace in the text?", "textwrap",
     "The text first has its whitespace collapsed."),
    ("Who can read the directory created by mkdtemp?", "tempfile",
     "readable, writable, and searchable only by the creating user"),
    ("Is a NamedTemporaryFile deleted automatically by default?", "tempfile",
     "whether the file is automatically deleted (default True)"),
    ("Why use compare_digest instead of comparing secrets with ==?", "secrets",
     "designed to prevent timing analysis"),
    ("What happens to the last batch of itertools batched when the input runs out?", "itertools",
     "The last batch may be shorter than n."),
    ("What does lru_cache do when maxsize is None?", "functools",
     "the LRU features are disabled and the cache can grow without bound"),
    ("Which characters replace plus and slash in urlsafe_b64encode?", "base64",
     "uses '-' instead of '+' and '_' instead of '/'"),
    ("Does the star wildcard of glob match filenames that start with a dot?", "glob",
     "filenames starting with a dot are special cases that are not matched by '*' and '?'"),
    ("Does gzip compress store the modification time of the data?", "gzip",
     "The modification time is set to 0 by default, for reproducibility."),
    ("What does math comb return when k is greater than n?", "math",
     "evaluates to zero when k > n"),
    ("What rational number does Fraction(8, 6) produce?", "fractions",
     "Fraction(8, 6) will produce a rational number equivalent to 4/3"),
]
DOMANDE = [g[0] for g in GOLDEN]
ANCORE = [g[2] for g in GOLDEN]

The reference of each question is a literal string taken from the docstring, not a paraphrase, and this lets the golden set verify itself.

documenti = costruisci_corpus()
by_id = {d["id"]: d for d in documenti}
for domanda, modulo, ancora in GOLDEN:
    assert normalizza(ancora) in normalizza(by_id[modulo]["testo"]), (modulo, ancora)
print(f"documents: {len(documenti)} | golden set questions: {len(GOLDEN)}")

The assert is the most useful part of the block: if a future version of Python changes a docstring, the golden set breaks loudly instead of silently becoming wrong. The golden set is a versioned file that depends on the version of Python you built it with, and it is worth declaring that.

From every question I derive two cases. The first, of family R, has the context that contains the answer: an 80-word window of the right docstring. The second, of family N, has the same question paired with the context of another module, and the right answer is abstention, that is, the exact string NOT IN CONTEXT. For the N case I do not take a random context: I take the other module's chunk that shares the most words with the question, because it is the hardest trap. A system that abstains only when the context is plainly unrelated has not learned to abstain.

STOP = set("a an and are as at be by for from if in into is it its of on or that the this to when with "
           "what which who how does do not can than then there their was were will".split())

def parole(testo):
    testo = re.sub(r"([a-z])([A-Z])", r"\1 \2", testo)
    return [w for w in re.findall(r"[a-z0-9]+", testo.lower()) if w not in STOP]

def finestra_fissa(testo, dim, sovrapposizione):
    parole_ = testo.split()
    passo = max(1, dim - sovrapposizione)
    return [" ".join(parole_[i:i + dim]) for i in range(0, len(parole_), passo) if parole_[i:i + dim]] or [testo]

def frasi(testo):
    return [f for f in re.split(r"(?<=[.!?])\s+", normalizza(testo)) if f]

def modello_finto(domanda, span, soglia=2):
    """Scripted extractor: it is NOT a model. Returns (answer, chosen sentence)."""
    qt = set(parole(domanda))
    punti, scelta = 0, ""
    for frase in frasi(span):
        p = len(qt & set(parole(frase)))
        if p > punti:
            punti, scelta = p, frase
    if punti < soglia:
        return "NOT IN CONTEXT", ""
    return scelta, scelta

These are the building blocks shared with the RAG lesson. parole removes the stop words and splits camelCase; finestra_fissa cuts the text into 80-word windows that slide by 64, so an answer straddling a boundary ends up whole in at least one window; modello_finto chooses the sentence with the most terms of the question and abstains below the threshold. The second value it returns, the chosen sentence, is used only by the label.

def costruisci_casi():
    casi = []
    for i, (domanda, modulo, ancora) in enumerate(GOLDEN):
        pezzi = finestra_fissa(by_id[modulo]["testo"], 80, 16)
        buono = next(p for p in pezzi if normalizza(ancora) in normalizza(p))
        casi.append({"id": f"R{i + 1:02d}", "domanda": domanda, "contesto": buono,
                     "atteso": ancora, "risponde": True})
        # family N: the chunk of ANOTHER module that shares the most words with the question
        # and does not contain the answer (the hardest trap, not a random one)
        qt = set(parole(domanda))
        candidati = [(len(qt & set(parole(p))), -k, p)
                     for k, d in enumerate(documenti) if d["id"] != modulo
                     for p in finestra_fissa(d["testo"], 80, 16)
                     if normalizza(ancora) not in normalizza(p)]
        casi.append({"id": f"N{i + 1:02d}", "domanda": domanda, "contesto": max(candidati)[2],
                     "atteso": "NOT IN CONTEXT", "risponde": False})
    return casi

CASI = costruisci_casi()
print(f"cases: {len(CASI)} ({sum(c['risponde'] for c in CASI)} answerable, "
      f"{sum(not c['risponde'] for c in CASI)} abstention)")
print(f"context words per case: median {statistics.median(len(c['contesto'].split()) for c in CASI)}")

The selection of contexts for family N looks at all the windows of the other modules and takes the one with the most words in common with the question, excluding those that contain the anchor; on a tie the first one wins, thanks to the second element of the tuple. The output says how many cases are born and how long the context is.

cases: 36 (18 answerable, 18 abstention)
context words per case: median 80.0

The cost of this golden set was low because the answer is a string written in a docstring, and the labels are decided by construction. When the answer requires an expert, the arithmetic changes: every case costs the time of a person who knows, and that time is the real limit of any evaluation. I declared one defect that I did not fix: a few anchors, such as harmonic_mean([40, 60]) 48.0, are code examples and not sentences, and the scripted systems get them wrong by construction. It is one question out of 18: it weighs 5.6% of family R, and I leave it in, declared, because it is the order of magnitude of the labelling noise of a real golden set.

The three systems and the missing label

To evaluate you need a label for every answer: right or wrong. With real models you would assign it by reading, one by one. Here systems A and A2 are programs, so I know which sentence they copied from and I can decide by rule: an answer of family R is right if the source sentence contains all the words of the reference, and one of family N is right if the system abstains.

def sistema_estrattore(caso, soglia=2):
    """Returns (answer, source sentence). The provenance is used only by the label."""
    return modello_finto(caso["domanda"], caso["contesto"], soglia)

def etichetta(caso, risposta, origine):
    """Reference label (1 = correct). In real life you give it, by reading; here the
    program can give it because the system is scripted and the source sentence is known."""
    astenuta = risposta.strip() == "NOT IN CONTEXT"
    if not caso["risponde"]:
        return int(astenuta)
    return int(not astenuta and set(parole(caso["atteso"])) <= set(parole(origine)))

The label is an oracle, and it is not allowed in a real case: inside this lesson it serves to fix the truth against which I measure the metrics and the judges. For system C the truth cannot be deduced, and I write it myself next to each answer.

# System C: it is NOT a program, it is a table of answers written by me, with my label
# next to each. It produces what an extractor never produces: right answers with
# words different from the reference, and a few plausible but wrong answers.
RIFORMULATE = {
    "R01": ("Matches scoring below 0.6 are ignored unless you change the cutoff.", 1),
    "R02": ("The insertion index lands in front of the first equal item already stored.", 1),
    "R03": ("The new item goes after the last equal one.", 1),
    "R04": ("It does both steps in one go, which is faster than pushing then popping.", 1),
    "R05": ("An empty dataset makes it raise StatisticsError.", 1),
    "R06": ("The average speed is 50 km/hr.", 0),
    "R07": ("The smaller of the two central values.", 1),
    "R08": ("Runs of whitespace are squeezed into single spaces before cutting.", 1),
    "R09": ("Only the user who created it can read, write or enter it.", 1),
    "R10": ("No, you must delete it yourself.", 0),
    "R11": ("It takes the same time whatever the inputs, so timing attacks learn nothing.", 1),
    "R12": ("Every batch has exactly n items.", 0),
    "R13": ("Nothing is ever evicted, so the cache grows without limit.", 1),
    "R14": ("Dash stands in for plus and underscore stands in for slash.", 1),
    "R15": ("No, names that begin with a dot are skipped by the star.", 1),
    "R16": ("Yes, the current time is stored in the header.", 0),
    "R17": ("It gives zero.", 1),
    "R18": ("The fraction four thirds.", 1),
    "N03": ("The new item goes after the last equal one.", 0),     # invents an answer where it should have abstained
    "N09": ("Only the user who created it can read it.", 0),
}

def sistema_riformulatore(caso):
    return RIFORMULATE.get(caso["id"], ("NOT IN CONTEXT", 1))

The twenty rows contain four plausible and wrong answers in family R (R06 says 50 km/h instead of 48, R10 says the file does not delete itself, R12 says every batch has exactly n items, R16 says gzip stores the current time) and two hallucinations in family N, where the system answers with a sentence that the context does not contain. The other fourteen R answers are right and none of them reuses the words of the reference: that is the property I need for the section on metrics.

USCITE = {"A": [sistema_estrattore(c) for c in CASI],
          "A2": [sistema_estrattore(c, 3) for c in CASI]}
RISPOSTE = {n: [r for r, _ in u] for n, u in USCITE.items()}
VERO = {n: [etichetta(c, r, o) for c, (r, o) in zip(CASI, u)] for n, u in USCITE.items()}
RISPOSTE["C"] = [sistema_riformulatore(c)[0] for c in CASI]
VERO["C"] = [sistema_riformulatore(c)[1] for c in CASI]
print(f"{'system':<8}{'R (18)':>8}{'N (18)':>8}{'total':>8}")
for nome, v in VERO.items():
    r = sum(x for x, c in zip(v, CASI) if c["risponde"]); n = sum(v) - r
    print(f"{nome:<8}{r:>8}{n:>8}{sum(v):>6}/36")

system    R (18)  N (18)   total
A              8      14    22/36
A2             5      17    22/36
C             14      16    30/36

How to read the table: A and A2 have the same total accuracy, 22 out of 36, but not the same right cases. A2 abstains more and gains on family N (from 14 to 17), losing on family R (from 8 to 5). The mean makes them equal, and the section on regressions shows why they are not. C scores 30 out of 36 and balances the two families: it errs four times where it should have answered and twice where it should have kept quiet.

Thirty-six cases are not enough to rank two systems, and you can compute it

The accuracy on a golden set is the success frequency of n binary trials: a proportion, with its uncertainty. The Wald interval, p̂ ± z·√(p̂(1 − p̂)/n), on small samples can leave [0, 1] and covers less than the declared 95%. You use the Wilson interval, which pulls the centre toward 0.5 by an amount that vanishes as n grows and widens the radius to account for the discreteness of the count.

from scipy.stats import norm

def wilson(successi, n, confidenza=0.95):
    z = norm.ppf(1 - (1 - confidenza) / 2)
    p = successi / n
    centro = (p + z * z / (2 * n)) / (1 + z * z / n)
    raggio = (z / (1 + z * z / n)) * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return centro - raggio, centro + raggio

def n_necessario(p1, p2, alpha=0.05, potenza=0.80):
    """Two independent proportions, two-sided test: n per group."""
    z_a, z_b = norm.ppf(1 - alpha / 2), norm.ppf(potenza)
    pm = (p1 + p2) / 2                          # common proportion under the null hypothesis
    num = (z_a * np.sqrt(2 * pm * (1 - pm))
           + z_b * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
    return int(np.ceil(num / (p1 - p2) ** 2))

print("Wilson 95% for an observed accuracy of 0.70")
print(f"{'n':>6}{'lower':>11}{'upper':>11}{'width':>10}{'step 1/n':>12}")
for n in (20, 36, 100, 400, 1000, 4000):
    lo, hi = wilson(round(0.70 * n), n)
    print(f"{n:>6}{lo:>11.4f}{hi:>11.4f}{hi - lo:>10.4f}{1 / n:>12.4f}")
print("n per system, two independent proportions, 5% two-sided, power 80%")
for p2 in (0.75, 0.80, 0.90):
    print(f"  0.70 vs {p2:.2f}  (difference {p2 - 0.70:.2f})  n = {n_necessario(0.70, p2)}")
for nome in ("A", "C"):
    s = sum(VERO[nome]); lo, hi = wilson(s, len(CASI))
    print(f"system {nome}: {s}/{len(CASI)} = {s / len(CASI):.3f}  Wilson 95% [{lo:.4f}, {hi:.4f}]")

The second function answers the question upstream: given two true rates, how many cases per system are needed for the two-sided test, at 5%, to find the difference 80% of the time. The term pm is the common proportion under the null hypothesis; without it you would use the same variance even when the two rates are far apart.

Wilson 95% for an observed accuracy of 0.70
     n      lower      upper     width    step 1/n
    20     0.4810     0.8545    0.3735      0.0500
    36     0.5314     0.8200    0.2885      0.0278
   100     0.6042     0.7811    0.1769      0.0100
   400     0.6534     0.7428    0.0895      0.0025
  1000     0.6709     0.7276    0.0567      0.0010
  4000     0.6856     0.7140    0.0284      0.0003
n per system, two independent proportions, 5% two-sided, power 80%
  0.70 vs 0.75  (difference 0.05)  n = 1251
  0.70 vs 0.80  (difference 0.10)  n = 294
  0.70 vs 0.90  (difference 0.20)  n = 62

With twenty questions and an observed accuracy of 0.70 the interval runs from 0.4810 to 0.8545: a system at 48% and one at 85% are both compatible with the measurement. The "step" is 1/n, the smallest difference the golden set can express: at twenty cases it is 0.05, so a 5% improvement is not small, it is the smallest that exists, with an uncertainty seven times larger than itself. To tell 0.70 from 0.75 you need 1,251 cases per system; for 0.70 against 0.90, 62 are enough.

system A: 22/36 = 0.611  Wilson 95% [0.4486, 0.7522]
system C: 30/36 = 0.833  Wilson 95% [0.6811, 0.9213]

My two systems, measured on the 36 cases, have Wilson intervals of 0.4486-0.7522 (A) and 0.6811-0.9213 (C): they overlap by about seven points (from 0.68 to 0.75). Read as two means, they settle nothing. This is the rule to take away: decide the smallest difference you want to be able to read and derive n from that, not the other way round. If you care about 5% and the budget gives you thirty-six cases, you do not have a small golden set, you have an instrument that does not measure what you want to measure.

The paired comparison, and when it does not help

The 1,251 questions per system hold for two independent samples. But A and C answer the same questions, so the difficulty of the cases is shared and can be removed. McNemar's test looks only at the discordant cases, those in which one system is right and the other is not: under the hypothesis that the two systems are equivalent, every discordant case is a fair coin, and the test becomes an exact binomial test. In parallel I compute a bootstrap interval of the difference, resampling the cases and not the rows: the pair stays together, and that is what makes it paired.

from scipy.stats import binomtest

def discordanti(x, y):
    return (sum(1 for a, b in zip(x, y) if a and not b),     # only the first is right
            sum(1 for a, b in zip(x, y) if b and not a))     # only the second is right

def mcnemar_esatto(x, y):
    n10, n01 = discordanti(x, y)
    # the concordant pairs do not enter: under the null hypothesis each discordant is a fair coin
    return n10, n01, binomtest(n10, n10 + n01, 0.5).pvalue if n10 + n01 else 1.0

def bootstrap_differenza(x, y, ripetizioni=10_000, seme=7):
    """95% percentile interval of the accuracy difference, resampling the CASES
    (the pair x[i], y[i] stays together: this is what makes the comparison paired)."""
    rng = np.random.default_rng(seme)
    d = np.array(x) - np.array(y)
    idx = rng.integers(0, len(d), size=(ripetizioni, len(d)))
    medie = d[idx].mean(axis=1)
    return float(d.mean()), float(np.percentile(medie, 2.5)), float(np.percentile(medie, 97.5))

n10, n01, p = mcnemar_esatto(VERO["A"], VERO["C"])
conc = len(CASI) - n10 - n01
print(f"A vs C: concordant {conc}, only A {n10}, only C {n01}, exact McNemar p = {p:.4f}")
d, lo, hi = bootstrap_differenza(VERO["A"], VERO["C"])
print(f"difference A - C = {d:+.3f}, paired bootstrap 95% [{lo:+.3f}, {hi:+.3f}]")
sa, sb = sum(VERO["A"]), sum(VERO["C"])
print(f"Wilson A [{wilson(sa, 36)[0]:.3f}, {wilson(sa, 36)[1]:.3f}] and C [{wilson(sb, 36)[0]:.3f}, {wilson(sb, 36)[1]:.3f}]: they overlap")

A vs C: concordant 22, only A 3, only C 11, exact McNemar p = 0.0574
difference A - C = -0.222, paired bootstrap 95% [-0.417, -0.028]
Wilson A [0.449, 0.752] and C [0.681, 0.921]: they overlap

Of the 36 cases, 22 are concordant and carry no information. In the 14 discordant ones C is right 11 times and A 3: an 11 to 3 outcome on a fair coin has an exact probability of p = 0.0574, just above 5%. The paired bootstrap instead gives an interval for the difference that excludes zero, from −0.417 to −0.028. The two instruments look at the same data and land on opposite sides of the threshold: it is the honest answer of a measurement at its limit, and it must not be resolved by choosing the instrument that agrees with you. With 14 discordant cases the percentile bootstrap tends to be optimistic, but that is a general caveat that I have not verified here with a dedicated simulation.

The question remains whether pairing always pays off. I try it with simulated systems on 100 cases, where I control how much of the difficulty is shared between the two.

def simula_sistemi(n, a1, a2, rigidita, rng):
    """Two SIMULATED systems on the same n cases: every case has a shared latent
    difficulty; `rigidita` says how much it weighs (0 = independent cases, 3 = highly shared)."""
    difficolta = rng.normal(size=n) * rigidita
    p = lambda a: 1 / (1 + np.exp(-(a - difficolta)))
    return (rng.random(n) < p(a1)).astype(int), (rng.random(n) < p(a2)).astype(int)

def potenza_simulata(n, a1, a2, rigidita, prove=1000, seme=11):
    rng = np.random.default_rng(seme)
    vinte_nonappaiato = vinte_appaiato = 0
    medie = np.zeros(2)
    for _ in range(prove):
        x, y = simula_sistemi(n, a1, a2, rigidita, rng)
        medie += (x.mean(), y.mean())
        pm = (x.mean() + y.mean()) / 2
        z = (x.mean() - y.mean()) / np.sqrt(2 * pm * (1 - pm) / n) if 0 < pm < 1 else 0.0
        vinte_nonappaiato += abs(z) > 1.96
        vinte_appaiato += mcnemar_esatto(x, y)[2] < 0.05
    return vinte_nonappaiato / prove, vinte_appaiato / prove, medie / prove

print("power over 1000 simulated trials, n = 100 cases, same two systems vs shared difficulty")
print(f"{'rigidity':>9}{'acc. 1':>8}{'acc. 2':>8}{'unpaired':>14}{'McNemar':>10}")
for rigidita in (0.0, 1.5, 3.0):
    a, b, (m1, m2) = potenza_simulata(100, 0.847, 1.386, rigidita)
    print(f"{rigidita:>9.1f}{m1:>8.3f}{m2:>8.3f}{a:>14.3f}{b:>10.3f}")

power over 1000 simulated trials, n = 100 cases, same two systems vs shared difficulty
 rigidity  acc. 1  acc. 2      unpaired   McNemar
      0.0   0.697   0.799         0.396     0.328
      1.5   0.645   0.730         0.220     0.257
      3.0   0.597   0.657         0.054     0.176

With independent cases (rigidity 0) the unpaired test wins by a little, 0.396 against 0.328, because the exact test is conservative. When the difficulty is shared the advantage flips: at rigidity 3 the unpaired test finds the difference only 5.4% of the time and McNemar 17.6%. The mean accuracies drop as rigidity grows, because the shared difficulty pushes everyone down, so the correct reading is: the advantage of pairing depends on how much the two systems err on the same cases, and in a real golden set you find that out by counting the concordant ones.

The price is a condition that is violated by distraction: the questions must be literally the same, in the same order, with the same context. If between the two evaluations even the retrieval changes, and in a RAG it changes at every reindexing, the comparison is no longer paired. This is why the golden set is frozen in a versioned file, with an identifier for each case.

Automatic metrics and vocabulary

Labelling by hand at every commit is not sustainable, and this is where automatic metrics come in, which compare the answer with the reference without understanding either. There are three, in order of strictness. The exact match is one if the two normalised strings coincide. The token F1 is the harmonic mean of precision and recall over the words in common, as in the SQuAD evaluation. The F1 on the longest common subsequence is the same harmonic mean on the LCS, the core of ROUGE-L, and unlike the previous one it takes word order into account.

import string
ARTICOLI = {"a", "an", "the"}

def norm_metrica(t):
    t = t.lower().translate(str.maketrans("", "", string.punctuation))
    return [w for w in t.split() if w not in ARTICOLI]

def f1_token(prev, rif):
    a, b = norm_metrica(prev), norm_metrica(rif)
    if not a or not b:
        return float(a == b)             # two empties are equal, one empty is not
    da = collections.Counter(b)
    comuni = 0
    for w in a:
        if da[w] > 0:                    # with multiplicity: three "the" do not count three times
            comuni += 1
            da[w] -= 1
    if comuni == 0:
        return 0.0                       # avoids 0/0 in the harmonic mean
    p, r = comuni / len(a), comuni / len(b)
    return 2 * p * r / (p + r)

def exact_match(prev, rif):
    return float(norm_metrica(prev) == norm_metrica(rif))

def f1_lcs(prev, rif):
    a, b = norm_metrica(prev), norm_metrica(rif)
    M = np.zeros((len(a) + 1, len(b) + 1), dtype=int)
    for i, x in enumerate(a, 1):
        for j, y in enumerate(b, 1):
            M[i, j] = M[i - 1, j - 1] + 1 if x == y else max(M[i - 1, j], M[i, j - 1])
    L = int(M[len(a), len(b)])
    if L == 0:
        return 0.0
    p, r = L / len(a), L / len(b)
    return 2 * p * r / (p + r)

The normalisation removes punctuation, capitals and articles, because "A list of paths." and "a list of paths" are the same answer. The count with multiplicity in the Counter stops an answer that repeats a word of the reference ten times from earning ten points of overlap, and the two return 0.0 before the harmonic mean avoid the division by zero when there is no token in common.

METRICHE = {"exact match": exact_match, "token F1": f1_token, "LCS F1": f1_lcs}
print(f"{'system':<9}{'metric':<13}{'mean':>7}{'agreement':>11}{'false neg.':>12}{'false pos.':>12}")
for nome in ("A", "C"):
    for mn, m in METRICHE.items():
        punti = [m(r, c["atteso"]) for r, c in zip(RISPOSTE[nome], CASI)]
        verdetto = [int(p >= 0.5) for p in punti]
        v = VERO[nome]
        fn = sum(1 for x, y in zip(verdetto, v) if y and not x)
        fp = sum(1 for x, y in zip(verdetto, v) if x and not y)
        accordo = sum(int(x == y) for x, y in zip(verdetto, v)) / len(v)
        print(f"{nome:<9}{mn:<13}{sum(punti) / len(punti):>7.3f}{accordo:>11.3f}{fn:>12}{fp:>12}")
print("RIGHT answers of system C with the lowest F1")
giuste_c = sorted((f1_token(r, c["atteso"]), c["id"], r, c["atteso"])
                  for c, r, v in zip(CASI, RISPOSTE["C"], VERO["C"]) if v and c["risponde"])
for f1, i, r, rif in giuste_c[:3]:
    print(f"{i}  F1 {f1:.3f}  exact match {exact_match(r, rif):.0f}")
    print(f"   reference: {rif}")
    print(f"   answer   : {r}")

system   metric          mean  agreement  false neg.  false pos.
A        exact match    0.417      0.806           7           0
A        token F1       0.601      0.972           1           0
A        LCS F1         0.601      0.972           1           0
C        exact match    0.444      0.611          14           0
C        token F1       0.547      0.667          12           0
C        LCS F1         0.537      0.639          13           0

The "agreement" is the fraction of answers on which the metric, with threshold 0.5, gives my own verdict. On system A the F1 reaches 0.972: a single error out of 36. On system C it drops to 0.667, with 12 false negatives and no false positives. The contrast is the lesson: lexical metrics are neither good nor bad, they depend on how the system writes. An extractor copies the reference, so the F1 judges it well; a system that rewords, with the same correctness, is punished.

RIGHT answers of system C with the lowest F1
R03  F1 0.000  exact match 0
   reference: insert it to the right of the rightmost x
   answer   : The new item goes after the last equal one.
R18  F1 0.000  exact match 0
   reference: Fraction(8, 6) will produce a rational number equivalent to 4/3
   answer   : The fraction four thirds.
R04  F1 0.080  exact match 0
   reference: runs more efficiently than heappush() followed by a separate call to heappop()
   answer   : It does both steps in one go, which is faster than pushing then popping.

Three right answers of system C have an F1 of 0.000, 0.000 and 0.080. "The fraction four thirds" is the right answer to "Fraction(8, 6) will produce a rational number equivalent to 4/3" and does not share a single word with the reference. The F1 does not measure correctness: it measures how much the answer resembles the way the author of the reference would have written it. It is still worth using when the right answer is a canonical string (an exception name, an error code, an identifier, a class label): there rewording is a bug and the exact match is the metric, not a fallback.

The automatic judge and agreement measured with Cohen's kappa

If the exact match is too strict and the F1 measures vocabulary, the next move is to ask a model to do what I did by hand: say whether the answer carries the same information as the reference. This is the automatic judge made popular by the work on MT-Bench and Chatbot Arena, which also discusses its biases of position, of verbosity and of preference for itself. The judge's prompt and the verdict parser are these.

GIUDICE = """You are grading a candidate answer against a reference answer.
QUESTION: {domanda}
REFERENCE: {atteso}
CANDIDATE: {risposta}
Does the candidate answer convey the same information as the reference?
Reply with exactly one word: CORRECT or WRONG."""

def verdetto(testo):
    t = testo.strip().upper()
    if re.search(r"\bINCORRECT\b|\bWRONG\b", t):
        return 0
    if re.search(r"\bCORRECT\b", t):
        return 1
    return None            # out of format: it must be counted, not silently converted

The verdict is requested as a single word, not inside a rationale, and the function returns None when the output is not in the requested format: that count must be reported, because a judge that is out of format in 20% of the cases is not strict, it is broken. The prompt is the template to adapt to your model and has not been run; the parser, instead, runs, and that is where you see its traps.

for uscita in ("CORRECT", "WRONG.", "Incorrect", "The answer is correct because...", "Sure! WRONG"):
    print(f"{uscita!r:<40} -> {verdetto(uscita)}")

Five typical outputs of a model, passed to the parser: two clean, one with punctuation, one that reasons and one with a courtesy prefix.

'CORRECT'                                -> 1
'WRONG.'                                 -> 0
'Incorrect'                              -> 0
'The answer is correct because...'       -> 1
'Sure! WRONG'                            -> 0

The last row is a case that the parser accepts and should not: "The answer is correct because..." becomes 1 even though the model started to reason instead of answering with one word. A parser that is too permissive hides a judge that does not respect the format; the countermeasure is to require that the output, once spaces and trailing punctuation are removed, be exactly one of the two words.

The judge's accuracy against my labels is not enough: on an unbalanced set, a judge that always says the majority class has high accuracy and zero information. You need Cohen's kappa, which measures agreement net of chance: κ = (p_o − p_e) / (1 − p_e), where p_o is the observed agreement and p_e is what you would get by combining the two marginal frequencies at random. A perfect judge has κ = 1, one indistinguishable from chance has κ = 0, one that systematically disagrees has a negative κ.

def rng_di(*chiavi):
    """One generator for EACH decision, derived from the judge's name and the case: same
    case, same judge, same verdict, in whatever order you call it."""
    return np.random.default_rng(_h("|".join(map(str, chiavi))))

def _h(s):
    return int.from_bytes(hashlib.blake2b(s.encode(), digest_size=8).digest(), "big")

class GiudiceSimulato:
    """NOT a model. Given the truth (which we pass it), it errs with fixed probabilities:
    false negatives partly higher when the answer shares few words with the reference
    (the typical error of someone judging the form), false positives at a fixed rate."""
    def __init__(self, nome, p_fn, extra_fn_forma, p_fp):
        self.nome, self.p_fn, self.extra, self.p_fp = nome, p_fn, extra_fn_forma, p_fp

    def verdetto(self, caso, risposta, vero):
        r = rng_di(self.nome, caso["id"], risposta).random()
        if vero:
            diversa = f1_token(risposta, caso["atteso"]) < 0.5
            return int(r >= self.p_fn + (self.extra if diversa else 0.0))
        return int(r < self.p_fp)

class GiudiceARegole:
    """A real, deterministic judge: CORRECT if the token F1 exceeds the threshold."""
    nome = "rule-based"
    def verdetto(self, caso, risposta, vero):
        return int(f1_token(risposta, caso["atteso"]) >= 0.5)

GIUDICI = [GiudiceSimulato("G1 pigro", 0.0, 0.0, 1.0),       # always says CORRECT
           GiudiceSimulato("G2 severo", 0.30, 0.30, 0.02),
           GiudiceSimulato("G3 buono", 0.08, 0.10, 0.05),
           GiudiceARegole()]

The delicate point is rng_di: every decision has its own generator, derived from the name of the judge, the case and the answer. It follows that the same judge gives the same verdict on the same input in whatever order you call it, and that adding a case does not move the verdicts of the others. A global generator with a single seed would instead have made every verdict depend on how many had been requested before. The false negatives of the simulated judge grow when the answer has few words in common with the reference: it is the typical error of someone judging the form. The names G1 pigro, G2 severo and G3 buono are Italian (lazy, strict, good) because they are part of that derivation, as I said at the start.

def kappa(y, x):
    n = len(y)
    po = sum(int(a == b) for a, b in zip(y, x)) / n
    pe = sum((y.count(c) / n) * (x.count(c) / n) for c in (0, 1))
    return (po - pe) / (1 - pe) if pe < 1 else 0.0

TUTTE = [(c, n, RISPOSTE[n][i], VERO[n][i]) for n in ("A", "A2", "C") for i, c in enumerate(CASI)]
y = [t[3] for t in TUTTE]
print(f"{'judge':<11}{'agreement':>10}{'kappa':>8}{'FN':>5}{'FP':>5}{'says CORRECT':>14}")
VERDETTI = {}
for g in GIUDICI:
    x = [g.verdetto(c, r, v) for c, _, r, v in TUTTE]
    VERDETTI[g.nome] = x
    fn = sum(1 for a, b in zip(x, y) if b and not a); fp = sum(1 for a, b in zip(x, y) if a and not b)
    print(f"{g.nome:<11}{sum(int(a == b) for a, b in zip(x, y)) / len(y):>10.3f}{kappa(y, x):>8.3f}"
          f"{fn:>5}{fp:>5}{sum(x) / len(x):>14.3f}")
print(f"{'mine':<11}{1.0:>10.3f}{1.0:>8.3f}{0:>5}{0:>5}{sum(y) / len(y):>14.3f}   ({len(y)} verdicts)")
print("accuracy attributed to the systems by each judge (and according to my labels)")
for g in GIUDICI:
    riga = {n: sum(VERDETTI[g.nome][k] for k, t in enumerate(TUTTE) if t[1] == n) / 36 for n in ("A", "A2", "C")}
    print(f"  {g.nome:<11}" + "  ".join(f"{n} {v:.3f}" for n, v in riga.items()))
print("  " + f"{'mine':<11}" + "  ".join(f"{n} {sum(VERO[n]) / 36:.3f}" for n in ("A", "A2", "C")))

judge       agreement   kappa   FN   FP  says CORRECT
G1 pigro        0.685   0.000    0   34         1.000
G2 severo       0.667   0.399   36    0         0.352
G3 buono        0.843   0.659   13    4         0.602
rule-based      0.870   0.730   14    0         0.556
mine            1.000   1.000    0    0         0.685   (108 verdicts)

Row by row. G1 always says CORRECT: agreement 0.685 but κ = 0.000, because 68.5% of the answers are really right and always saying "right" hits that share without looking at anything. G2 is strict, 36 false negatives out of 108, with κ = 0.399. G3 has κ = 0.659. The rule-based judge, which is a real program, has the highest agreement (0.870) and the highest kappa (0.730). If you stopped here you would choose it.

accuracy attributed to the systems by each judge (and according to my labels)
  G1 pigro   A 1.000  A2 1.000  C 1.000
  G2 severo  A 0.333  A2 0.306  C 0.417
  G3 buono   A 0.500  A2 0.556  C 0.750
  rule-based A 0.583  A2 0.583  C 0.500
  mine       A 0.611  A2 0.611  C 0.833

The second table changes the verdict. What matters for deciding is the order of the systems, and the rule-based judge puts C (0.500) below A (0.583) when according to my labels C is worth 0.833 and A 0.611. Its kappa is high because most of the cases are literal extractions, where the F1 works fine; 12 of its 14 false negatives sit on system C, the one that rewords. G3, less accurate in absolute terms, has the right order (A 0.500, C 0.750). An error concentrated on one system cannot be read in the aggregate kappa, and is unmasked only by looking at the distribution of errors by system. The absolute levels are in any case incomparable: the same system C is worth 0.833 for me, 0.750 for G3, 0.417 for G2 and 1.000 for G1.

First bias: position

The most common way of having a system judged is the pairwise comparison: two answers to the same question and the choice of the better one. It is easier for the judge and has a defect that is measured in a few minutes: the verdict may depend on the order in which the answers are pasted into the prompt. It is one of the results of the work by Wang and colleagues on LLMs as referees, where the order of presentation was enough to make one model or the other win. Here I try it on a simulated judge, to show how it is measured.

def sig(x):
    return 1 / (1 + math.exp(-x))

class GiudiceDiCoppia:
    """SIMULATED pairwise comparison. It picks the better answer with ability `abilita`
    (in logits per point of truth difference), plus an advantage `beta` for whoever sits
    in position A and an advantage `bonus` for the answer of its own system."""
    def __init__(self, nome, abilita=2.5, beta=0.0, proprio=None, bonus=0.0):
        self.nome, self.abilita, self.beta, self.proprio, self.bonus = nome, abilita, beta, proprio, bonus

    def sceglie_A(self, caso, in_a, in_b, vero_a, vero_b):
        logit = self.abilita * (vero_a - vero_b) + self.beta
        logit += self.bonus * ((in_a == self.proprio) - (in_b == self.proprio))
        return int(rng_di(self.nome, caso["id"], in_a, in_b).random() < sig(logit))

def due_ordini(g, caso, x, y, vero_x, vero_y):
    v_ab = g.sceglie_A(caso, x, y, vero_x, vero_y)           # x in position A
    v_ba = g.sceglie_A(caso, y, x, vero_y, vero_x)           # now y sits in position A
    # WARNING: in the second pass "A" means the OTHER system. The same letter in the
    # two passes = the judge chose two different systems on the same pair.
    ribaltato = (v_ab == 1) == (v_ba == 1)
    vittoria_x = (v_ab + (1 - v_ba)) / 2
    return ribaltato, v_ab, v_ba, vittoria_x

The line that matters is the comment in due_ordini. In the second pass the letter A no longer indicates the same system, so a verdict A in both passes means that the judge chose two different systems on the same pair: it is a contradiction, not a confirmation. Whoever forgets the swap counts contradictions as agreements and publishes a flip rate close to zero. The ability is 2.5 logits per point of difference in the truth, and the position advantage (beta) is 0 for the neutral judge, +1.2 for one that prefers the first position and −1.2 for one that prefers the second.

COPPIE = [(i, c) for i, c in enumerate(CASI) if VERO["A"][i] != VERO["C"][i]]
print(f"pairs A vs C with different truth: {len(COPPIE)}/{len(CASI)} "
      f"(A better in {sum(VERO['A'][i] > VERO['C'][i] for i, _ in COPPIE)}, "
      f"C better in {sum(VERO['C'][i] > VERO['A'][i] for i, _ in COPPIE)})")
print(f"{'judge':<22}{'flipped':>10}{'first pos.':>12}{'A, AB only':>12}{'A, averaged':>13}")
for g in (GiudiceDiCoppia("neutro", beta=0.0), GiudiceDiCoppia("primacy", beta=1.2),
          GiudiceDiCoppia("recency", beta=-1.2)):
    ris = [due_ordini(g, c, "A", "C", VERO["A"][i], VERO["C"][i]) for i, c in COPPIE]
    rib = sum(r[0] for r in ris); prima = sum(r[1] + r[2] for r in ris)
    print(f"{g.nome:<22}{rib:>7}/{len(ris)}{prima:>9}/{2 * len(ris)}"
          f"{sum(r[1] for r in ris) / len(ris):>12.3f}{sum(r[3] for r in ris) / len(ris):>13.3f}")
print(f"wins of A according to my labels: {sum(VERO['A'][i] > VERO['C'][i] for i, _ in COPPIE) / len(COPPIE):.3f}")

def tasso_ribaltamento_atteso(abilita, beta, differenza=1):
    """Probability of contradicting itself on a pair, in closed form (no sampling)."""
    p_ab = sig(abilita * differenza + beta)               # picks A with x in A
    p_ba = sig(-abilita * differenza + beta)              # picks A with y in A
    return p_ab * p_ba + (1 - p_ab) * (1 - p_ba)
print("expected flip rate on a pair with different truth")
for beta in (0.0, 0.5, 1.2, 2.0):
    print(f"  beta {beta:.1f}  ->  {tasso_ribaltamento_atteso(2.5, beta):.3f}")

pairs A vs C with different truth: 14/36 (A better in 3, C better in 11)
judge                    flipped  first pos.  A, AB only  A, averaged
neutro                      0/14       14/28       0.286        0.286
primacy                     5/14       19/28       0.500        0.321
recency                     3/14       11/28       0.214        0.321
wins of A according to my labels: 0.214

The useful pairs are 14 out of 36, those in which A and C have different labels: in the other 22 the comparison has no real winner. According to my labels A wins 21.4% of the time. The neutral judge gives it 28.6%, both in a single pass and averaging. The judge with an advantage for the first position gives A 50.0% reading the AB pass alone, that is, it declares it level with C; averaging the two orders the value drops to 0.321, close to the truth. The one with an advantage for the second position does the opposite: 0.214 in one pass, 0.321 averaged. A single pass would have changed the conclusion in both directions. The judge's name neutro means neutral, and like the Italian names above it feeds the seed.

expected flip rate on a pair with different truth
  beta 0.0  ->  0.140
  beta 0.5  ->  0.155
  beta 1.2  ->  0.228
  beta 2.0  ->  0.380

The second table is the closed form: with ability 2.5 and no bias a judge contradicts itself anyway on 14% of the pairs, because its choice is probabilistic, and the rate rises to 38% with beta = 2. A practical rule follows: the flip rate has a floor that is not zero, and a value of exactly zero on few pairs is more suspicious than reassuring (here the neutral judge has 0 out of 14, a coincidence compatible with an expected value of about two). A pairwise comparison is always done in both orders, and it costs twice the calls.

Verbosity and self-preference: two biases to look for, not to inherit

The other two biases of the MT-Bench work have a different shape. Verbosity is the tendency to reward longer answers at equal content, so much so that AlpacaEval introduced a length-controlled version. Self-preference is the tendency of a model to give higher scores to its own texts, which Panickssery and colleagues related to the model's ability to recognise them. Both were measured on real models, and I cannot repeat those measurements here: I can show the experimental design and check that it is able to find a bias when there is one.

For verbosity: a correct answer is lengthened with true, off-topic sentences, and you watch how the score from 1 to 5 changes as the length grows. The slope is estimated per doubling of length, and the uncertainty by resampling the cases, not the rows, because the six versions of the same answer are not independent. The seed string of each run is built as f"verbosita{gamma}", again in Italian, for the reason given at the start.

ZEPPE = ["Only the name of this class is considered a public API.",
         "A simple framework for writing line-oriented command interpreters.",
         "The default timer is a fast built-in one based on real time.",
         "An ABC can be subclassed directly, and then acts as a mix-in class.",
         "The standard debugger class is an example of this design."]

def voto_1_5(nome, caso, testo, vero, gamma):
    """SIMULATED scoring judge: base 4 if correct and 2 if wrong, plus gamma for each
    doubling of length, plus noise. gamma > 0 = rewards whoever writes more."""
    parole_ = max(1, len(testo.split()))
    grezzo = 2 + 2 * vero + gamma * math.log2(parole_ / 10) + rng_di(nome, caso["id"], testo).normal(0, 0.5)
    return float(min(5, max(1, round(grezzo))))

def pendenza_con_intervallo(gamma, nome, ripetizioni=2000, seme=3):
    giuste = [(c, RISPOSTE["C"][i]) for i, c in enumerate(CASI) if c["risponde"] and VERO["C"][i]]
    punti = []
    for c, r in giuste:
        for q in range(6):                               # only the length grows
            t = " ".join([r] + ZEPPE[:q])
            punti.append((len(t.split()), voto_1_5(nome, c, t, 1, gamma), c["id"]))
    lunghezze = np.log2([p[0] for p in punti]); voti = np.array([p[1] for p in punti])
    pendenza = np.polyfit(lunghezze, voti, 1)[0]
    ids = sorted({p[2] for p in punti})
    rng = np.random.default_rng(seme)
    stime = []
    for _ in range(ripetizioni):                         # the CASES are resampled, not the rows
        scelti = set(rng.choice(ids, size=len(ids)).tolist())
        mask = np.array([p[2] in scelti for p in punti])
        stime.append(np.polyfit(lunghezze[mask], voti[mask], 1)[0])
    return len(giuste), pendenza, np.percentile(stime, 2.5), np.percentile(stime, 97.5)

print("judge's score per doubling of length (right answers + true, off-topic filler)")
print(f"{'true gamma':>11}{'cases':>6}{'slope':>10}{'95% CI':>20}")
for gamma in (0.0, 0.6, -0.4):
    n, p, lo, hi = pendenza_con_intervallo(gamma, f"verbosita{gamma}")
    print(f"{gamma:>11.1f}{n:>6}{p:>10.3f}   [{lo:+.3f}, {hi:+.3f}]")

judge's score per doubling of length (right answers + true, off-topic filler)
 true gamma cases     slope              95% CI
        0.0    14     0.050   [-0.031, +0.159]
        0.6    14     0.364   [+0.284, +0.435]
       -0.4    14    -0.491   [-0.603, -0.380]

With gamma = 0 the estimated slope is 0.050 and the interval [−0.031; +0.159] contains zero: no bias, which is the right answer. With gamma = +0.6 (rewards length) the slope is 0.364 and the interval excludes zero; with gamma = −0.4 it is −0.491 and this too excludes it, on the opposite side. The protocol finds both signs. It does not find the exact value of gamma: the slope is smaller in absolute value because the scores are integers and saturate at 1 and at 5. Since the sign is found and the size is not, the protocol is suited to answering "is it there or not", and not to estimating how large it is.

For self-preference you need a comparison between two judges: the one that evaluates its own system and a third-party judge that evaluates the same one. The bias is the gap between the two win rates, averaged over the two orders. Here the useful question is not how large it is, but from how many pairs upward you are able to see it.

def scarti_per_coppia(bonus, n_coppie, seme):
    """For each synthetic pair: the win rate (averaged over the two orders) that a judge
    with `bonus` for its own system gives it, minus that of a third-party judge."""
    rng = np.random.default_rng(seme)
    d = rng.choice([-1, 1], size=n_coppie)               # +1: the own system is the better one
    def sceglie_A(b, proprio_in_a):
        segno = 1 if proprio_in_a else -1
        logit = 2.5 * segno * d + segno * b              # the bonus pushes toward the own system
        return rng.random(n_coppie) < 1 / (1 + np.exp(-logit))
    def tasso(b):                                        # the own system wins: picks A if it sits in A,
        return (sceglie_A(b, True) + ~sceglie_A(b, False)) / 2   # does not pick A if it sits in B
    return tasso(bonus) - tasso(0.0)

def rilevato(scarti):
    """z on the mean of the gaps: the test you would really run on your data."""
    sd = scarti.std(ddof=1)
    return sd > 0 and abs(scarti.mean() / (sd / math.sqrt(len(scarti)))) > 1.96

print("own judge minus third-party judge, win rate averaged over both orders, 400 trials per row")
print(f"{'true bonus':>11}{'n pairs':>10}{'mean gap':>14}{'detected':>10}")
for bonus in (0.0, 0.5, 1.0):
    for n in (14, 140):
        prove = [scarti_per_coppia(bonus, n, s) for s in range(400)]
        print(f"{bonus:>11.1f}{n:>10}{np.mean([p.mean() for p in prove]):>14.3f}"
              f"{np.mean([rilevato(p) for p in prove]):>10.3f}")

own judge minus third-party judge, win rate averaged over both orders, 400 trials per row
 true bonus   n pairs      mean gap  detected
        0.0        14         0.001     0.005
        0.0       140        -0.001     0.045
        0.5        14         0.020     0.025
        0.5       140         0.020     0.245
        1.0        14         0.047     0.077
        1.0       140         0.047     0.765

With 14 pairs, the size I have available, a bias of 0.5 logits is detected in 2.5% of the trials and one of 1.0 in 7.7%: practically never. With 140 pairs, 1.0 logits is seen 76.5% of the time, 0.5 logits 24.5%. Where the bonus is null the test gives a false positive 4.5% of the time at 140 pairs, in line with the nominal 5%. The result I would report from a real measurement on 14 pairs is not "there is no self-preference", it is "I see none, and with this sample size I would not have seen even one of medium size". It is the difference between absence of evidence and evidence of absence, and it is declared next to the number.

Reproducibility: the seed does not make the judge stable

My judges are deterministic by construction, because the generator derives from the decision. A real judge is not by default, and the first move is to set the temperature to zero and fix the seed where the API allows it. I did not measure in this lesson how much a real model varies between one repetition and the next, and I do not publish a figure: it is a property of the model, of the version and of the hardware. The scheme to measure it on your judge is simple and has not been run: twenty judging prompts, ten repetitions each, and the count of the prompts with at least one minority verdict.

Determinism, even when it is there, is not stability of judgement. The position judge with beta = 1.2 is perfectly reproducible and still contradicts itself on one pair in five in expected value. Reproducible and reliable are two distinct properties, and the first is the easiest to confuse with the second.

Regressions between two versions, and a gate that sees them

The everyday use of a golden set is not comparing two different systems: it is comparing yesterday's system with today's. A and A2 have the same accuracy, 22 out of 36, and a mean cannot see it, but the two do not behave the same way.

def confronta_versioni(vecchia, nuova, critici):
    migliorati = [c["id"] for c, a, b in zip(CASI, VERO[vecchia], VERO[nuova]) if b and not a]
    regressioni = [c["id"] for c, a, b in zip(CASI, VERO[vecchia], VERO[nuova]) if a and not b]
    media, lo, hi = bootstrap_differenza(VERO[nuova], VERO[vecchia])
    critiche = [i for i in regressioni if i[0] in critici]
    passa = not critiche and lo >= -0.05
    return migliorati, regressioni, media, lo, hi, critiche, passa

for famiglia, etichetta_f in (("R", "answerable"), ("N", "abstention")):
    ok = lambda n: sum(v for v, c in zip(VERO[n], CASI) if c["id"][0] == famiglia)
    print(f"family {famiglia} ({etichetta_f}): A {ok('A')}/18  ->  A2 {ok('A2')}/18")
mig, reg, media, lo, hi, critiche, passa = confronta_versioni("A", "A2", critici="R")
print(f"total: A {sum(VERO['A'])}/36  ->  A2 {sum(VERO['A2'])}/36  (difference {media:+.3f}, 95% CI [{lo:+.3f}, {hi:+.3f}])")
print(f"improved {len(mig)}: {' '.join(mig)}")
print(f"regressions {len(reg)}: {' '.join(reg)}")
print(f"regressions on critical cases (family R): {len(critiche)} -> gate {'PASSES' if passa else 'FAILS'}")

family R (answerable): A 8/18  ->  A2 5/18
family N (abstention): A 14/18  ->  A2 17/18
total: A 22/36  ->  A2 22/36  (difference +0.000, 95% CI [-0.139, +0.139])
improved 3: N07 N14 N16
regressions 3: R03 R05 R15
regressions on critical cases (family R): 3 -> gate FAILS

A2 improved three cases of family N (N07, N14, N16, where it now correctly abstains) and worsened three of family R (R03, R05, R15, where it used to answer well and now keeps quiet). The bootstrap interval of the total difference is [−0.139; +0.139]: perfectly centred. The gate rule looks elsewhere: no regression on the critical cases and a lower limit of the interval not below −0.05. Here the critical cases are family R, and A2 loses three of them, so the gate fails. It is the most useful form of a golden set in production: a threshold on the mean would let through the version that broke three right answers, and the list of regressed cases tells whoever receives the report where to start.

The cost of an evaluation round

With real models the cost of an evaluation is in seconds and tokens, and it decides whether it enters a development cycle or stays a quarterly ritual. I do not measure times here, because a scripted program takes milliseconds and says nothing about a model. I measure instead something you can carry with you: how many calls and how many prompt words are needed.

prompt_punto = [GIUDICE.format(domanda=c["domanda"], atteso=c["atteso"], risposta=r)
                for c, _, r, _ in TUTTE]
par_parole = [len(p.split()) for p in prompt_punto]
print(f"point-by-point round: {len(prompt_punto)} calls, {sum(par_parole)} prompt words "
      f"(median {int(statistics.median(par_parole))})")
print(f"pairwise round in both orders: {2 * len(COPPIE)} calls")

point-by-point round: 108 calls, 6239 prompt words (median 54)
pairwise round in both orders: 28 calls

A point-by-point round over the three systems, 108 answers, requires 108 calls and about 6,200 prompt words, with a median of 54 words per call; the pairwise comparison in both orders adds 28. To convert words into tokens and time you use the tokenizer and the speed of your judge (the lesson on tokenisation shows why the ratio changes from one language to another). The bill multiplies by the judges and by the orders: three judges on one more system make three times the calls. The right order is the golden set first, then a small judge, and the large one only where the small one does not tell them apart.

The protocol that holds the pieces together

1. Decide the minimum difference you want to read: from that you derive n, not the other way round.
2. Build the golden set and freeze it, with an identifier for each case, in a versioned file.
3. Include the cases in which the answer does not appear: they measure abstention and cost little.
4. Label a sample by hand: it is the calibration of everything else.
5. Measure the judge's agreement with accuracy, kappa and the share of "correct", and look at the errors by system.
6. Measure the biases on your task: position in both orders, length, a third-party judge.
7. Compare in a paired way, never two absolute means, and keep the list of the regressed cases.

The order matters, which is why this is a numbered list: each step makes the next one readable. Steps 1 and 7 are the ones that get skipped, and they are the two that decide whether the evaluation produces information. Step 4 is the only one that costs human effort, and it is redone at every change of model or of judge prompt: the calibration belongs to the judge-prompt pair, not to the judge.

Trade-offs

• Large golden set versus curated golden set. Size buys resolution, from a width of 0.3735 at twenty cases to 0.0567 at a thousand, but only if the labels stay good. Mitigation: grow in stages, relabelling a sample at every stage.
• Lexical metric versus automatic judge. The F1 costs microseconds and is reproducible, but on the system that rewords it gave an agreement of 0.667 against 0.972 on the extractor. The judge understands rewording and carries biases. Mitigation: F1 where the answer is canonical, judge where it is prose, and the kappa measured to know how far to trust it.
• Point by point versus pairwise. Point by point gives one number per system and one call per answer; pairs are easier for the judge but cost twice the calls and give only a ranking. Mitigation: point by point for monitoring, pairs to decide between two close candidates.
• Aggregate versus per system. The aggregate kappa of the rule-based judge is the highest and its ordering of the systems is wrong. Mitigation: always report the errors broken down by system and by family of cases.
• Evaluating at every commit versus evaluating well. The quick round against regressions has the cost of a hook; two orders, several judges and the bias tests cost many times as much. Mitigation: two declared regimes, the quick one at every commit and the full one before a decision.

The trade-offs have a common root: every shortcut moves the cost from a visible place (labelling time) to an invisible one (a systematic error that nobody measured). None of the five mitigations removes the cost; they turn it into a number you can read and discuss, and that is all you can ask of an evaluation.

The traps you really fall into

1. Comparing two means instead of the paired cases. Symptom: two accuracies that differ by a few points and a decision taken anyway. Cause: the intervals overlap, like my two from 36 cases. Remedy: the discordant cases, the exact test and the paired interval, read together.
2. A pairwise comparison in one order only. Symptom: the verdict changes when you swap the two blocks in the prompt. Cause: the position bias. Remedy: two passes and an average; if the flips are much above the expected floor, change the judge instead of averaging harder.
3. Forgetting that in the second pass the letter A indicates the other system. Symptom: a flip rate close to zero and a win rate that is too good. Cause: contradictory verdicts counted as confirmations. Remedy: the swap made explicit in the code, with a comment.
4. Reporting the judge's accuracy without the kappa. Symptom: "our judge agrees with 75% of the human judgements". Cause: on an unbalanced set that 75% can come from a judge that always says the same thing. Remedy: accuracy, kappa and the share of the positive class, always together.
5. Trusting the aggregate kappa. Symptom: the best judge on paper ranks the systems badly. Cause: the errors are concentrated on one system. Remedy: the accuracy attributed to each system, compared with the labels.
6. A golden set without negative cases. Symptom: high accuracy in evaluation and invented answers in production. Cause: all the questions had the answer in the context. Remedy: family N and the check that the right answer did not end up by chance in the wrong context.
7. Changing the judge's prompt and keeping the old calibration. Symptom: a jump in the numbers after a change that did not concern the system. Cause: the kappa had been measured with another prompt. Remedy: version the judge's prompt together with the golden set.
8. Concluding "no bias" from few cases. Symptom: "we checked, the judge does not prefer itself". Cause: with few pairs an average effect cannot be seen. Remedy: declare the sample size and the smallest detectable size next to the result.

The first four are errors of procedure and are avoided with a checklist; the last four are errors of interpretation and are avoided only by declaring, next to every number, what that number was able to see. It is worth keeping them at hand as questions to ask whoever presents an evaluation, yourself included.

Redoing the experiment

The code blocks of this page, in the order in which they appear and without changes, form a single program of about 530 lines: Python 3.12 or later with NumPy and SciPy, no network, no model. Paste them into a file, run it twice and compare the outputs with cmp: they must be identical, because every random decision derives from a seed and a case identifier. The block with the judge's prompt is a template to adapt and was run only in the part that calls no models, that is, the verdict parser.

What you take away

You now know how to size a golden set from the difference you want to read, how to compute the Wilson interval and the number of cases per system, how to compare two systems on the same cases with McNemar and with a paired bootstrap without trusting only one of them, and how to recognise a lexical metric that punishes whoever rewords. You know how to calibrate a judge with accuracy, kappa and the share of "correct", you know that a high kappa does not guarantee the right ordering of the systems, and you know how to design an experiment that looks for the bias of position, of verbosity and of self-preference, declaring how large an effect you would be able to see.

What this lesson does not give you is the size of the biases of a real model: the judges were simulated, and the results on their behaviour are properties of the simulator. For that size you need the measurements of the papers cited below, or your own, made with your data. Before asking a system whether it has improved, ask your yardstick how far it can be wrong: you know how to compute that number, and without it all the others are anecdotes. The next lesson, on running a model locally, takes the problem to where the judges are real: memory, quantisation and latency.

Sources

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv:2306.05685) - the automatic judge, the three biases (position, verbosity, self-preference) and the agreement with human preferences; abstract read, consulted on 2026-11-12.
• Large Language Models are not Fair Evaluators (arXiv:2305.17926) - the order in which answers are presented alters the verdict, and the calibration by balanced position; abstract read, consulted on 2026-11-12.
• LLM Evaluators Recognize and Favor Their Own Generations (arXiv:2404.13076) - self-preference and its link with the recognition of one's own texts; abstract read, consulted on 2026-11-12.
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (arXiv:2404.04475) - the length bias and the correction by regression; abstract read, consulted on 2026-11-12.
• SQuAD: 100,000+ Questions for Machine Comprehension of Text (arXiv:1606.05250) - the origin of exact match and token F1 as answer metrics; title verified, consulted on 2026-11-12.
• scipy.stats.binomtest (SciPy documentation) - the exact binomial test used for McNemar; page reached, consulted on 2026-11-12.

Wilson (1927), McNemar (1947) and Cohen (1960) remain bibliographic citations of the methods I implement: I did not reopen the original papers, and the calculations are verified by the blocks that were run and by the exercises solved by hand.