09 - Testing and Evaluation: Metrics and Benchmarks

The problem: the agent that passes today and fails tomorrow

A normal test says that a function, given an input, always returns the same output. With an agent that is not true: if you repeat the same request the model may choose different tools, in a different order, and succeed on Monday and fail on Tuesday.

Checking the agent once and declaring it working is the most common mistake, and the fix is not a bigger test but three kinds of checks with different purposes: deterministic tests on your code, with the model replaced by a script; statistical repetitions on the real behavior, to know how reliable it is and not just whether it can succeed; continuous checks in production, to notice when something changes. This article puts them in that order, with listings run without calling any model.

The vocabulary is that of Anthropic's guide “Demystifying evals for AI agents”: a trial is a single run of a task; a grader is the logic that assigns a score; the transcript (also called trace or trajectory) is the complete record of a trial, with outputs, tool calls and intermediate results; the outcome is the final state of the environment. The distinction between transcript and outcome is the first thing to understand: an agent that says “flight booked” has produced a sentence, and whether the booking really exists is told only by the final state.

First level: testing the trajectory with a fake model

The part of an agent that is your code (the loop, the caps, the tools, the validation) is tested deterministically by replacing the model with a script of moves.

The listing defines a minimal agent for booking a flight: the “model” is a list of moves (tool calls or a final answer), the tools are local functions that modify a state, and checks applies four verifications, two on the outcome (there is a single booking, the right one), two on the trajectory (search before booking, no repeated call) and one on the limit (the sequence ends with an answer). Three scripts, one correct and two faulty, show what each check sees.

def run_agent(script, max_steps=5):
    """Minimal agent: the 'model' is a script of moves. Returns the trace and the outcome."""
    state = {"bookings": []}
    def search_flight(destination):
        return {"id": "V1", "to": destination}
    def book(flight_id):
        state["bookings"].append(flight_id)
        return "ok"
    tools = {"search_flight": search_flight, "book": book}
    trace = []
    for step, move in enumerate(script):
        if step >= max_steps:
            trace.append(("interrupted", "step cap"))
            break
        if "final" in move:
            trace.append(("final", move["final"]))
            break
        trace.append((move["tool"], move["arguments"]))
        tools[move["tool"]](**move["arguments"])
    return trace, state

def checks(trace, state):
    calls = [t[0] for t in trace]
    return {
        "outcome: exactly one booking of V1": state["bookings"] == ["V1"],
        "trajectory: search before booking": calls[:2] == ["search_flight", "book"],
        "trajectory: no repeated call": len(calls) == len(set(map(str, trace))),
        "limit: ends with an answer": trace[-1][0] == "final",
    }

SEARCH = {"tool": "search_flight", "arguments": {"destination": "Lisbon"}}
BOOK = {"tool": "book", "arguments": {"flight_id": "V1"}}
CASES = {
    "correct": [SEARCH, BOOK, {"final": "Booked."}],
    "double booking": [SEARCH, BOOK, {**BOOK, "arguments": {"flight_id": "V1"}}, {"final": "Done."}],
    "never ends": [SEARCH] * 8,
}
for name, script in CASES.items():
    outcome = checks(*run_agent(script))
    print(f"{name}: {sum(outcome.values())}/{len(outcome)}", [k for k, v in outcome.items() if not v])

# Output:
# correct: 4/4 []
# double booking: 2/4 ['outcome: exactly one booking of V1', 'trajectory: no repeated call']
# never ends: 0/4 ['outcome: exactly one booking of V1', 'trajectory: search before booking', 'trajectory: no repeated call', 'limit: ends with an answer']

The correct script passes everything. The one with the double booking passes “search before booking” and the limit, but is stopped by two checks: the outcome (two bookings, not one) and the repeated call. The third never ends, and the step cap interrupts it: had I checked only the final answer, an agent that does not conclude would not even have been noticed.

The design point is that the costliest errors are not in the text of the answer, they are in the state of the world after the run. These checks run in milliseconds, with no model cost, and are the only ones you can run on every change to the code. They say nothing about the choices a real model would make: for those you need repetitions.

Second level: repeating, because a single run does not measure

Anthropic's guide explains why: every task has its own success rate (perhaps 90% on one and 50% on another) and a task that succeeded in one trial may fail in the next. So you need several trials per task and two metrics. pass@k is the probability of getting at least one correct solution in k attempts, and it grows with k;

pass^k is the probability that all k trials succeed, and it falls with k. The guide gives the example of an agent with a 75% success rate per trial: over three trials the probability of passing all of them is about 42%. The τ-bench benchmark introduced pass^k to measure reliability and reports, for the agents tried in its study, that in the “retail” domain the value at k=8 falls below 25%.

The listing simulates three tasks with a known success probability (0.9, 0.5 and 0.1) and a generator with a fixed seed, runs ten trials each, and computes the two metrics with the combinatorial formula: if among n trials there are c successes, the probability that k trials chosen at random all succeed is C(c,k)/C(n,k). At the end it computes the Wilson interval for a proportion, the method that the NIST handbook presents as recommended by Brown, Cai and DasGupta, to show how wide the uncertainty is.

import random
from math import comb, sqrt
from statistics import NormalDist

TASK = {"A": 0.9, "B": 0.5, "C": 0.1}          # success probability of the fake agent, per task
N = 10                                          # trials per task
rng = random.Random(2026)                       # fixed seed: same output on every run
successes = {t: sum(rng.random() < p for _ in range(N)) for t, p in TASK.items()}

def pass_at_k(c, n, k):                         # at least one success among k trials drawn from the n
    return 1 - comb(n - c, k) / comb(n, k)

def pass_all_k(c, n, k):                        # all k succeed
    return comb(c, k) / comb(n, k)

print("task  successes pass@1 pass@3 pass^3 pass^8")
for t, c in successes.items():
    print(f"{t:8} {c:2}/{N}   {c/N:.2f}   {pass_at_k(c, N, 3):.2f}   {pass_all_k(c, N, 3):.2f}   {pass_all_k(c, N, 8):.2f}")
tot = sum(successes.values())
print("mean pass@1:", round(tot / (N * len(TASK)), 2),
      "| mean pass^8:", round(sum(pass_all_k(c, N, 8) for c in successes.values()) / len(TASK), 2))

def wilson(c, n, level=0.95):
    z = NormalDist().inv_cdf(0.5 + level / 2)
    p = c / n
    center = (p + z * z / (2 * n)) / (1 + z * z / n)
    margin = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return round(center - margin, 2), round(center + margin, 2)

print("90% observed over 10 trials :", wilson(9, 10))
print("90% observed over 100 trials:", wilson(90, 100))

# Output:
# task  successes pass@1 pass@3 pass^3 pass^8
# A        10/10   1.00   1.00   1.00   1.00
# B         3/10   0.30   0.71   0.01   0.00
# C         1/10   0.10   0.30   0.00   0.00
# mean pass@1: 0.47 | mean pass^8: 0.33
# 90% observed over 10 trials : (0.6, 0.98)
# 90% observed over 100 trials: (0.83, 0.94)

Task B has a true probability of 0.5 but gave 3 successes out of 10: with ten trials that is the noise, and that is why a “before and after” comparison made on a few runs says little. The pass@3 column rewards versatility (task B reaches 0.71) while pass^3 punishes it (0.01): which one matters to you depends on the use, and for an agent facing a customer, which must succeed every time, it is the second.

The average goes from 0.47 to a pass^8 of 0.33 with the same data. The last two computations show that an observed frequency of 90% over 10 trials is compatible with a true value between 0.60 and 0.98, while over 100 trials the interval narrows to 0.83-0.94. A warning: repeated trials on the same task are not independent of each other the way different tasks would be, so the interval is an order of magnitude of the uncertainty, not a guarantee.

One consequence of this reasoning concerns the size of the dataset. Anthropic's guide again argues that you do not need hundreds of tasks to start: 20-50 simple tasks drawn from real failures are a good beginning, because in the early stages every change has a large effect and small samples are enough; more mature agents require larger sets to detect smaller effects. It also names two kinds of sets: capability tasks, with a low success rate at the start, and regression tasks, which must stay close to 100% and signal that something broke when they drop.

A model as a judge: useful, but it must be checked

For free-text answers there is no exact result to compare against, and grading them by hand does not scale. So a model is used as a judge. The paper “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena” examines the use and the limits of this approach, among them position bias, verbosity bias and self-preference bias, and reports that a strong judge such as GPT-4 agrees with human preferences in more than 80% of cases, the same level of agreement as between people, on their two benchmarks.

These are the authors' measurements on that data: in your domain they must be verified. Anthropic's guide reports the practice of a team that moved from manual grading to model-based graders, with criteria defined by the product and periodic human calibration, and recommends reading the transcripts to understand whether a failure belongs to the agent or to the grader.

The listing calls no model: it tests two fake, faulty judges with the same checks you would apply to a real one. The first always picks the answer it sees first, the second picks the longest. For each pair of answers it asks for the verdict in both orders (A then B, B then A): a reliable judge must pick the same answer in both. Then it compares the verdicts with the preference of a person, using the percentage of agreement and Cohen's kappa coefficient, which removes the part of the agreement due to chance.

# (answer A, answer B, which one a person prefers)
PAIRS = [
    ("Restart the service.", "Restart the service, then check the logs and verify the proxy configuration.", "B"),
    ("Renew the certificate.", "The certificate seems to have expired, so the first thing to do is to check the expiry date and then, if needed, renew it with the intended command.", "A"),
    ("Upgrade the library to version 2.", "No action is needed.", "A"),
    ("It depends.", "It depends on the load: above the threshold use the queue, otherwise a direct call.", "B"),
    ("Use the default password.", "Change the password and enable two-factor authentication.", "B"),
    ("The disk is full: free up some space.", "The disk is full, so the first thing is to consider evaluating whether it might be possible to free up space.", "A"),
]

def judge_position(first, second):              # always picks the first answer
    return "first"

def judge_length(first, second):                # prefers the longer answer
    return "first" if len(first) >= len(second) else "second"

def verdict(judge, a, b, order):
    first, second = (a, b) if order == "AB" else (b, a)
    choice = judge(first, second)
    return ("A" if choice == "first" else "B") if order == "AB" else ("B" if choice == "first" else "A")

def kappa(x, y):
    n = len(x)
    observed = sum(a == b for a, b in zip(x, y)) / n
    expected = sum((x.count(c) / n) * (y.count(c) / n) for c in set(x) | set(y))
    return round((observed - expected) / (1 - expected), 2) if expected < 1 else 1.0

for name, judge in (("always the first", judge_position), ("the longest", judge_length)):
    forward = [verdict(judge, a, b, "AB") for a, b, _ in PAIRS]
    reverse = [verdict(judge, a, b, "BA") for a, b, _ in PAIRS]
    human = [h for _, _, h in PAIRS]
    consistent = sum(f == r for f, r in zip(forward, reverse))
    print(f"{name:16} consistent if I swap the order: {consistent}/6 | "
          f"agreement with the person: {sum(f == h for f, h in zip(forward, human))}/6 | kappa: {kappa(forward, human)}")

# Output:
# always the first consistent if I swap the order: 0/6 | agreement with the person: 3/6 | kappa: 0.0
# the longest      consistent if I swap the order: 6/6 | agreement with the person: 4/6 | kappa: 0.33

The judge that always picks the first answer has zero consistency, and the check with the swapped order exposes it at once: its agreement with the person (3 out of 6) is purely by chance, kappa 0. The judge that picks the longest is perfectly consistent (6 out of 6) and has a kappa of 0.33: consistency is not enough, because a judge can be wrong in the same way every time.

It is wrong in the two cases in which the better answer is the shortest (the second and the sixth). That is why a calibration set must also contain pairs in which the right answer is the concise one: without them, a judge that measures only length looks good. The pairs in the listing are written on purpose and the numbers are those of an example: they serve to show the method, not to evaluate a model.

Beyond accuracy: cost, latency, reliability

An agent that succeeds but costs ten times more than necessary, or that times out now and then, is not a good agent. The paper that proposes the CLEAR framework (Cost, Latency, Efficacy, Assurance, Reliability) argues that current benchmarks mostly evaluate completion accuracy and neglect cost, reliability and operational stability.

It reports, on the authors' data, cost variations of up to 50 times at equal accuracy, a performance that drops from 60% with a single trial to 25% with consistency over eight trials, and agents optimized only for accuracy that are 4.4 to 10.8 times more expensive than cost-aware alternatives with comparable performance; an evaluation by 15 experts would have given a correlation with production success of 0.83 for CLEAR against 0.41 for accuracy alone.

These are claims of a single research group, on 300 enterprise tasks of their own, and I did not reproduce them; I report them for the idea, not for the figures. You can apply the idea without the framework: record for each trial the outcome, the number of steps, the tokens and the time (the trace in Multi-Agent Orchestration already contains them), and compare two versions of the agent on all four columns, not just the first.

Public benchmarks: what they are for

Public benchmarks measure general capabilities on tasks chosen by others. Five examples, in alphabetical order:

• AgentBench: eight different environments to evaluate the reasoning and decisions of models used as agents;
• GAIA: 466 questions that require reasoning, web use and tools, of which the answers to 300 are kept private for the leaderboard;
• SWE-bench: 2,294 problems taken from real issues of 12 popular Python repositories, in which the agent modifies the code; SWE-bench Lite is a subset of 300 more self-contained tasks;
• τ-bench: conversations with a simulated user, tools and domain rules, evaluated by comparing the final state of the database with the expected one;
• WebArena: working websites in four domains (commerce, forums, collaborative development, content management) with long-horizon tasks.

The order here is alphabetical and says nothing about importance, but there is a difference that matters: SWE-bench and τ-bench evaluate the final state (project tests, database), so they measure an outcome; GAIA checks a short answer.

I do not report the leaderboards because they change and the figures in the original papers capture the models of that time (for example WebArena reported 14.41% for the best agent against 78.24% for people, and GAIA 15% against 92%). They serve to choose a model or to compare two approaches on a task similar to yours; they do not replace your own set of tasks, which must come from your own failures.

Comparing two versions and noticing drift

When you change a prompt or a model you want to know whether the new version is better, and how many trials it takes to say so. For two proportions the sample size per group depends on the starting rate, the difference you want to detect, the significance level and the power. The listing implements the standard formula with the normal distribution of the standard library and applies it to an improvement of five points from three starting rates. Then it uses the same test for a drift check: it compares a reference window with the last week.

from math import ceil, sqrt
from statistics import NormalDist

N = NormalDist()

def trials_per_group(p1, p2, alpha=0.05, power=0.80):
    """Trials for each variant to tell p1 from p2 (two-sided test, two proportions)."""
    z_a, z_b = N.inv_cdf(1 - alpha / 2), N.inv_cdf(power)
    variance = p1 * (1 - p1) + p2 * (1 - p2)
    return ceil((z_a + z_b) ** 2 * variance / (p1 - p2) ** 2)

print("base success -> +5 points: trials per variant")
for base in (0.50, 0.80, 0.90):
    print(f"  {base:.2f} -> {base + 0.05:.2f}: {trials_per_group(base, base + 0.05)}")

def z_two_proportions(c1, n1, c2, n2):
    p = (c1 + c2) / (n1 + n2)
    error = sqrt(p * (1 - p) * (1 / n1 + 1 / n2))
    z = (c1 / n1 - c2 / n2) / error
    return round(z, 2), f"p = {2 * (1 - N.cdf(abs(z))):.2g}"

# Drift check: reference window against the last week
print("reference 180/200 against 170/200    :", z_two_proportions(180, 200, 170, 200))
print("reference 1800/2000 against 1700/2000:", z_two_proportions(1800, 2000, 1700, 2000))

# Output:
# base success -> +5 points: trials per variant
#   0.50 -> 0.55: 1562
#   0.80 -> 0.85: 903
#   0.90 -> 0.95: 432
# reference 180/200 against 170/200    : (1.51, 'p = 0.13')
# reference 1800/2000 against 1700/2000: (4.78, 'p = 1.7e-06')

The same five-point jump requires 1,562 trials per variant when starting from 0.50, 903 from 0.80 and 432 from 0.90: “you need about 1,500 tasks” is true only for the first case, and that is why the figure must not be repeated without the starting rate.

In the drift check, a drop from 90% to 85% over 200 trials per window has p = 0.13, so it cannot be told apart from chance; with the same drop over 2,000 trials per window p is 1.7 times ten to the minus six. The decision on the alarm threshold depends on how many false alarms you are willing to handle. The test assumes independent trials and a single comparison: looking at it every day without correcting increases false reports, and repeating the same tasks breaks independence.

In production, the traces and the metrics of the system (latency per agent, tokens per request, circuit breaker openings) described in the articles on orchestration and tools feed this check. If the agent has a memory, the retrievals must be tested too, with tasks in which the right answer is in the retrieved chunks and others in which it is not, as in Memory Systems. Hostile cases (injection, out-of-scope requests) have their own set of tests, covered in Security and Safety. Every failure observed in production becomes a new task in the set, as Anthropic's guide suggests by starting from real failures.

Frequently asked questions

What you can do now

You can deterministically test an agent's loop by replacing the model with a script and checking outcome, trajectory and limits. You can repeat trials and read pass@k, pass^k and a confidence interval without mistaking three successes out of ten for a property of the agent. You can verify an automatic judge with the order swap and with the agreement against a person, compute how many trials you need to compare two versions, and tell a drift from a random fluctuation. You also know what public benchmarks are for, and that your own set of tasks must be born from your own failures.

Repeated trials cost tokens, and how to set a spending cap per task is in FinOps: Cost Optimization for Agents and LLM Tokens; what you have measured is then taken into operation as described in production deployment.