86 - Structured Output: Getting Data You Can Validate Out of an Agent

As long as an agent answers in prose, whoever sits downstream has to guess: pull a number out with a regex, work out whether "yes, I'd say go ahead" means approved. With structured output you declare the type of the answer and you get only two outcomes: an object that respects the schema, or an explicit error. It is not a formatting convenience: it is the point where an agent stops being a chatbot and becomes a component you can plug into another piece of software. Here you see how to declare it, what the SDK really does when the answer does not fit, how to recover, and what the schema does not guarantee.

What I ran, and what I did not. Code tested on 2026-11-12 with openai-agents 0.24.0, pydantic 2.14.3 and Python 3.14.4, in a temporary virtual environment. No model was called, not even Ollama: the model is fake and answers with a script I wrote, through a simulated HTTP transport (no network, no port). So the outputs below prove what the SDK and Pydantic do with a given answer, not how well a real model fills a schema. Every Python listing was run twice with identical output. The names model, read_stock and read_incoming_orders come from the lessons 09-04 and 09-05; in the listings I import them from a lessons module of your own, and in the test model() returns the fake model. Pin the SDK version: the names in this API have changed between series.

The analogy: the medical prescription versus the pharmacist's advice

A pharmacist who tells you "for that pain I'd take an anti-inflammatory, maybe after meals" gives you useful information that a system cannot process. An electronic prescription has mandatory fields: drug code, dosage, number of packs, instructions. The same content, but the second form can be transmitted, stored, validated and rejected if it is incomplete. Structured output is the step from advice to prescription.

The two downstream architectures, compared

The difference is not aesthetic: it changes where the uncertainty ends up. With prose, the uncertainty moves into your parsing code and becomes an intermittent bug. With the declared type, the uncertainty stays at the boundary and becomes an explicit error you can handle.

The mechanism: a Pydantic type instead of a string

The parameter is output_type of Agent (imported from agents). It accepts a Pydantic model, but the SDK documentation says that any type you can wrap in a TypeAdapter will do: dataclasses, lists, TypedDict. If you do not write it, it is None and the agent produces text: final_output is a string. Here I use a Pydantic model because the constraints (Literal, ge, max_length) become part of the schema and of the validation with the same declaration. The result is that the SDK does three things: it generates the JSON schema of the type, sends it to the model together with the request, and validates the answer before handing it to you.

import asyncio
from typing import Literal
from pydantic import BaseModel, Field
from agents import Agent, Runner
from lessons import model, read_stock, read_incoming_orders   # defined in 09-04 and 09-05

class Decision(BaseModel):
    sku: str
    outcome: Literal["reorder", "do_not_reorder", "insufficient_data"]
    quantity: int = Field(ge=0, le=5000, description="0 if not reordering")
    reason: str = Field(max_length=200)
    data_used: list[str] = Field(description="Names of the tools actually consulted")

agent = Agent(
    name="Reorder decider",
    instructions=("Decide whether to reorder. Read the data with the tools. "
                  "If a piece of data is missing, use outcome='insufficient_data' and quantity=0."),
    tools=[read_stock, read_incoming_orders],
    output_type=Decision,
    model=model(),
)

def create_proposal(sku: str, quantity: int) -> None:
    print(f"order proposal: {quantity} x {sku}")      # placeholder: your management system goes here

async def main():
    result = await Runner.run(agent, "Should SKU-11 be reordered?")
    d = result.final_output                 # an instance of Decision, not a string
    print(type(d).__name__, d.outcome, d.quantity, "-", d.reason)
    if d.outcome == "reorder":
        create_proposal(d.sku, d.quantity)  # ordinary code, no parsing

asyncio.run(main())

# Output:
# Decision do_not_reorder 0 - 4 in stock but 20 incoming

The listing defines an agent that decides on a reorder with two tools (the ones from the previous lesson) and an output type Decision. The agent is tested against a script of three answers: it calls read_stock, calls read_incoming_orders, then answers with a JSON text. That text is written by me, not produced by a model: what the output shows is that result.final_output is an instance of Decision, not a string. The gain is in the last lines: d.outcome == "reorder" is a comparison between values of a Literal, not an interpretation of text, and d.quantity is an int in the declared range, not a string to convert and hope for the best. With the fake warehouse data (4 pieces, threshold 10, 20 incoming) the coherent decision is "do not reorder", and indeed create_proposal is not called.

What really travels to the model

The schema the SDK builds from the type can be inspected without any model. The listing continues in the same file, and prints the schema in "strict" mode, the default one.

import json
from agents import AgentOutputSchema

schema = AgentOutputSchema(Decision)
print("strict:", schema.is_strict_json_schema())
print(json.dumps(schema.json_schema(), indent=2, ensure_ascii=False))

# Output:
# strict: True
# {
#   "properties": {
#     "sku": {
#       "title": "Sku",
#       "type": "string"
#     },
#     "outcome": {
#       "enum": [
#         "reorder",
#         "do_not_reorder",
#         "insufficient_data"
#       ],
#       "title": "Outcome",
#       "type": "string"
#     },
#     "quantity": {
#       "description": "0 if not reordering",
#       "maximum": 5000,
#       "minimum": 0,
#       "title": "Quantity",
#       "type": "integer"
#     },
#     "reason": {
#       "maxLength": 200,
#       "title": "Reason",
#       "type": "string"
#     },
#     "data_used": {
#       "description": "Names of the tools actually consulted",
#       "items": {
#         "type": "string"
#       },
#       "title": "Data Used",
#       "type": "array"
#     }
#   },
#   "required": [
#     "sku",
#     "outcome",
#     "quantity",
#     "reason",
#     "data_used"
#   ],
#   "title": "Decision",
#   "type": "object",
#   "additionalProperties": false
# }

There are four things to read. First: required contains all the fields, and additionalProperties is false: this is the "strict" shape the SDK imposes. Second: the constraints minimum, maximum and maxLength are in the schema, so the model "sees" them; whether a server enforces them during generation is another matter (below). Third: the fields are in the order in which you wrote them in the class, and this matters for the reasoning field, later on. Fourth: in the HTTP request I read from the simulated transport, the schema is inside response_format with type: json_schema and strict: true, next to tools in the same request. The Ollama OpenAI-compatibility page lists response_format among the supported fields of /v1/chat/completions, and the "Structured outputs" page says a JSON schema can be enforced on the answers. I have not verified on a real Ollama whether a given model respects strict or ignores the numeric constraints.

What happens when the answer does not fit

The question that separates an example from a system is this: what does your code see if the model gets it wrong? The SDK validates with Pydantic in strict mode, and if that fails it raises ModelBehaviorError. I passed eight answers to the same validator (one valid and seven variants: six defective, one with an extra field), calling validate_json directly (it is the method the run uses on the model's text).

from agents import AgentOutputSchema
from agents.exceptions import ModelBehaviorError

schema = AgentOutputSchema(Decision)
OK = '{"sku": "SKU-11", "outcome": "reorder", "quantity": 250, "reason": "below threshold", "data_used": []}'
cases = {
    "valid":             OK,
    "out of range":      OK.replace("250", "9999"),
    "value not allowed": OK.replace('"reorder"', '"maybe"'),
    "missing field":     '{"sku": "SKU-11", "outcome": "reorder"}',
    "text in front":     "Here is the result: " + OK,
    "markdown fence":    "```json\n" + OK + "\n```",
    "number as text":    OK.replace("250", '"250"'),
    "extra field":       OK.replace("}", ', "note": "x"}'),
}
for name, text in cases.items():
    try:
        schema.validate_json(text)
        print(f"{name:20} -> accepted")
    except ModelBehaviorError as e:
        print(f"{name:20} -> ModelBehaviorError: {e.message}")

# Output:
# valid                -> accepted
# out of range         -> ModelBehaviorError: Invalid JSON when parsing model output
# value not allowed    -> ModelBehaviorError: Invalid JSON when parsing model output
# missing field        -> ModelBehaviorError: Invalid JSON when parsing model output
# text in front        -> ModelBehaviorError: Invalid JSON when parsing model output
# markdown fence       -> ModelBehaviorError: Invalid JSON when parsing model output
# number as text       -> ModelBehaviorError: Invalid JSON when parsing model output
# extra field          -> accepted

Two observations, both uncomfortable. The first: the message is always the same, "Invalid JSON when parsing model output", whatever the defect. The SDK does not tell you which field is wrong; in my test the exception raised by Runner.run did not even carry the run data (run_data was None). For a useful retry, you have to reconstruct that detail yourself. The second: text around the JSON and the markdown fence are rejected, because the answer must be only JSON; and an extra field is instead accepted, because Pydantic ignores extra fields unless you write model_config = ConfigDict(extra="forbid") (tested: with that option the extra field is rejected). "250" in quotes is also rejected: the validation is strict and does not convert strings to numbers. A warning on stderr such as "No active span; trace error was not attached" appears when you call validate_json outside a run: it is not part of the output and not an error in your code.

Constrained decoding: the mechanism, and where the guarantee stops

Several of the defects in the list of eight answers would be one problem fewer if the model could not get the shape wrong. A mechanism that prevents it exists, constrained decoding, and it is worth understanding, because it explains both its strength and its limits. What follows is a conceptual illustration, not the code of a server: I have not observed it on Ollama.

A language model generates one token at a time: at each step it produces a probability distribution over the whole vocabulary and then picks one token. Constrained decoding slots in between these two moments: before the choice, it zeroes the probability of the tokens that would make the text no longer completable into a valid JSON with respect to the schema. With a one-field schema, outcome, which can be "reorder" or "do_not_reorder", the steps are these.

state: ""                  allowed: only "{"            (the schema is an object)
state: "{"                 allowed: only "\""           (a key is needed)
state: "{\""               allowed: only tokens that start "outcome"
                                     (it is the only possible key)
state: "{\"outcome\""      allowed: only ":"
state: "{\"outcome\":"     allowed: only "\""
state: "{\"outcome\": \""  allowed: only tokens that start "reorder"
                                     or "do_not_reorder"
state: "...\"reorder"      allowed: only "\""           (the enum is complete)
state: "...\"reorder\""    allowed: only "}"            (no other field)

At each step the space of possibilities is reduced to what the schema allows. The model chooses within that subset: it can decide between "reorder" and "do_not_reorder", but it cannot emit "maybe", cannot forget the quotation mark and cannot add a field. The result is valid by construction, not by luck. Three consequences follow.

The server decides how much of the schema is applied. To keep the state of the grammar, a server usually supports only part of JSON Schema, and which part depends on it: read its documentation. What I observed myself sits one step earlier, in the SDK, and is in the listing below: a type that the strict schema cannot represent is rejected before anything is sent, with a UserError.

The shape is guaranteed, the content is not. The filter acts on syntactic validity: between "reorder" and "do_not_reorder" both are allowed, and the choice is made by the model with its probabilities. No decoding mechanism can know which answer is the right one. That is why Pydantic validation remains indispensable even where constrained decoding exists: constraints such as le=5000 may not be enforced by the server (I have not verified this on Ollama), and the SDK rechecks them anyway.

Constraining the format changes what the model can write. A model that, left free, would have written two sentences of reasoning before concluding, under constraint must emit the required field immediately. The remedy is to put a reasoning field before the decision fields: models generate left to right, so reasoning written first can influence what follows, whereas reasoning written afterwards is a justification. The order of the fields in the schema is that of the class (you saw it above); whether a given server generates the fields in that order I have not verified.

When the server does not constrain: tolerate and retry

With a small local model, or with a server that accepts the schema and then ignores it, constrained decoding may be missing, and a safety net is needed. The net has two parts: accepting a JSON surrounded by text ("Here is the result: ...", a markdown fence), and sending back to the model the precise reason for the rejection, which the SDK does not supply. The way I tested is a subclass of AgentOutputSchema that overrides validate_json: it uses the same TypeAdapter as the SDK, in strict mode, but first extracts the first balanced {...} object, and when it fails it remembers why.

import json
from agents import AgentOutputSchema, Runner
from agents.exceptions import ModelBehaviorError
from pydantic import ValidationError

def extract_json(text: str) -> str:
    """The first balanced {...} object; if there is none, the text as it is."""
    start = text.find("{")
    if start < 0:
        return text
    depth, in_string, skipped = 0, False, False
    for i in range(start, len(text)):
        c = text[i]
        if in_string:
            if skipped:
                skipped = False
            elif c == "\\":
                skipped = True
            elif c == '"':
                in_string = False
        elif c == '"':
            in_string = True
        elif c == "{":
            depth += 1
        elif c == "}":
            depth -= 1
            if depth == 0:
                return text[start:i + 1]
    return text

class TolerantSchema(AgentOutputSchema):
    """Like AgentOutputSchema, but extracts the JSON from the text and remembers why it rejected."""
    def __init__(self, type_):
        super().__init__(type_)
        self.last_error: str | None = None

    def validate_json(self, json_str: str):
        try:
            obj = self._type_adapter.validate_json(extract_json(json_str), strict=True)
        except ValidationError as e:
            self.last_error = "; ".join(
                f"{'.'.join(map(str, d['loc']))}: {d['msg']}" for d in e.errors())
            raise ModelBehaviorError("non-conforming output") from e
        self.last_error = None
        return obj

async def structured_with_recovery(agent, request, schema, attempts=2):
    message = request
    for _ in range(attempts):
        try:
            return (await Runner.run(agent, message)).final_output
        except ModelBehaviorError:
            message = (f"{request}\n\nThe previous answer was not valid ({schema.last_error}). "
                       "Reply ONLY with a JSON object that conforms to the schema.")
    raise ValueError(f"non-conforming output after {attempts} attempts: {schema.last_error}")

Three details. extract_json counts braces but ignores those inside strings, because a reason such as "below the threshold {10}" contains them. TolerantSchema remembers the error in an instance attribute: that is fine for an agent used sequentially, not for concurrent runs on the same object (in that case you need one schema per run). And structured_with_recovery catches ModelBehaviorError, not ValidationError: that is the exception the SDK raises, and catching the other would never catch anything. Here is what the diagnosis says on the same eight cases as before.

tolerant = TolerantSchema(Decision)
for name, text in cases.items():
    try:
        tolerant.validate_json(text)
        print(f"{name:20} -> accepted")
    except ModelBehaviorError:
        print(f"{name:20} -> rejected: {tolerant.last_error}")

# Output:
# valid                -> accepted
# out of range         -> rejected: quantity: Input should be less than or equal to 5000
# value not allowed    -> rejected: outcome: Input should be 'reorder', 'do_not_reorder' or 'insufficient_data'
# missing field        -> rejected: quantity: Field required; reason: Field required; data_used: Field required
# text in front        -> accepted
# markdown fence       -> accepted
# number as text       -> rejected: quantity: Input should be a valid integer
# extra field          -> accepted

Now the text around the JSON and the fence get through, while "out of range", "value not allowed", "missing field" and "number as text" are rejected with the reason. That reason is what makes the second attempt effective: the model corrects a specific field instead of retrying at random. I test it with a script of two answers: the first has quantity at 9999 (surrounded by text), the second is valid (with a courtesy sentence before and after).

import asyncio
from agents import Agent

schema = TolerantSchema(Decision)
tolerant_agent = Agent(name="Reorder decider", instructions="Decide whether to reorder.",
                       output_type=schema, model=model())

async def demo():
    d = await structured_with_recovery(tolerant_agent, "Should SKU-11 be reordered?", schema)
    print("recovered:", d.outcome, d.quantity)

asyncio.run(demo())

# Output:
# recovered: do_not_reorder 0

In the second request, read from the simulated transport, the user message was: "Should SKU-11 be reordered? The previous answer was not valid (quantity: Input should be less than or equal to 5000). Reply ONLY with a JSON object that conforms to the schema.". Every retry is one more whole run: two attempts are fine, five mean that the schema or the instruction is wrong, not the model. If the second also fails, the function raises ValueError with the last reason, and it is up to the caller to decide whether to send the case to review.

How to design a schema the model fills in well

The rules that follow come from the way the schema is built and from the way the SDK treats it; where I did not measure an effect on quality, I say so.

• Close the sets. Every field that can be an enum must be a Literal: the space of answers becomes finite and explicit. That this improves quality I have not measured; that it removes every out-of-list value from the answer, I have.
• Provide an "I don't know" state. Without a value such as insufficient_data, you force the model to choose among wrong options, and it will choose.
• Use field descriptions. They end up in the schema (you saw them above) and work as micro-prompts: "0 if not reordering" avoids a null field and an argument.
• Keep the schema flat. Fewer levels, fewer things to get wrong and less risk of exceeding what the server supports.
• Ask for a traceability field. A data_used: list[str] does not stop the model from lying, but it makes the lie comparable with the real record of tool calls.

The five rules have an order of cost, not of importance: the first two are one line of declaration, the third is a string, the fourth is a discipline, the fifth requires a comparison in code (I write it further on). Before trusting a type, though, look at what the SDK does with it: I passed five innocent-looking types to AgentOutputSchema.

import datetime, decimal, json
from pydantic import BaseModel
from agents import AgentOutputSchema
from agents.exceptions import UserError

class Deadline(BaseModel):
    when: datetime.datetime
class Price(BaseModel):
    amount: decimal.Decimal
class Defaults(BaseModel):
    note: str = ""
    n: int | None = None
class Mapping(BaseModel):
    stocks: dict[str, int]
class A(BaseModel):
    a: int
class B(BaseModel):
    b: str
class Either(BaseModel):
    x: A | B

for T in (Deadline, Price, Defaults, Mapping, Either):
    try:
        s = AgentOutputSchema(T).json_schema()
        props = {k: {c: v for c, v in p.items() if c != "title"} for k, p in s["properties"].items()}
        print(f"{T.__name__:10} required={s['required']} {json.dumps(props, ensure_ascii=False)}")
    except UserError as e:
        print(f"{T.__name__:10} UserError: {e}")

# Output:
# Deadline   required=['when'] {"when": {"format": "date-time", "type": "string"}}
# Price      required=['amount'] {"amount": {"anyOf": [{"type": "number"}, {"type": "string"}]}}
# Defaults   required=['note', 'n'] {"note": {"default": "", "type": "string"}, "n": {"anyOf": [{"type": "integer"}, {"type": "null"}]}}
# Mapping    UserError: Strict JSON schema is enabled, but the output type is not valid. Either make the output type strict, or wrap your type with AgentOutputSchema(YourType, strict_json_schema=False)
# Either     required=['x'] {"x": {"anyOf": [{"$ref": "#/$defs/A"}, {"$ref": "#/$defs/B"}]}}

What you read: a field with a default value (note: str = "") ends up in required anyway, so in the strict schema the model must emit all the keys, and an "optional" field becomes at most "nullable" (anyOf with null). A datetime becomes a string with format: date-time, and a Decimal becomes "number or string", that is a type that does not tell you which of the two you will receive: for money, an integer number of cents or a string with a declared format is better. A dict[str, int] is rejected by the SDK, because in strict mode the keys must be known; if you really need it, there is the option AgentOutputSchema(Type, strict_json_schema=False), which gives up the guarantee. Unions of models (A | B) are accepted and produce anyOf with $defs, but I do not know whether your server handles them: check its documentation before using them.

A complete case: from the mail to the object

Structured extraction from free text is the most profitable use of this technique, because it replaces repetitive human work with something verifiable. Take an order-request mail written the way real customers write it, that is badly: "12 pcs", "SKU 11" without a hyphen, "about ten" which is a vague number, and two lines with different urgencies. The schema must anticipate all of them, instead of hoping.

import asyncio, re
from typing import Literal
from pydantic import BaseModel, Field
from agents import Agent, Runner
from lessons import model

MAIL = """
Good morning,
as discussed on the phone yesterday we would need 12 pcs of SKU 11 (the grey ones)
and if possible about ten of the 42s too. There is no rush for the 42s.
Ship to Via Roma 3, Lecce. Invoice Brambilla as usual.
Thanks
Marco
"""

class Row(BaseModel):
    reasoning: str = Field(max_length=300,
        description="Which sentence of the mail this row comes from. Quote it.")
    raw_sku: str = Field(description="The code EXACTLY as it appears in the text")
    quantity: int | None = Field(description="Exact number, or null if the text is vague")
    quantity_text: str | None = Field(description="The original wording, e.g. 'about ten'")
    urgency: Literal["standard", "urgent", "unspecified"]

class Extraction(BaseModel):
    rows: list[Row]
    shipping_address: str | None
    billing_customer: str | None
    uncertain_fields: list[str] = Field(
        description="Names of the fields you are NOT sure about. Empty list if everything is clear.")

extractor = Agent(
    name="Order extractor",
    instructions=(
        "Extract the order rows from the mail. Non-negotiable rules:\n"
        "1. Do not normalise codes: report them as they appear.\n"
        "2. If a quantity is vague ('about ten', 'some'), set quantity=null "
        "and put the wording in quantity_text. Do NOT guess a number.\n"
        "3. Every row must quote the sentence it comes from.\n"
        "4. List in uncertain_fields everything you have doubts about."
    ),
    output_type=Extraction,
    model=model(),
)

def normalize_sku(raw: str) -> str | None:
    """Deterministic: 'SKU 11', 'sku-11' and '11' become 'SKU-11'. Anything else: None."""
    m = re.fullmatch(r"(?:sku[\s_-]*)?0*(\d+)", raw.strip(), flags=re.IGNORECASE)
    return f"SKU-{m.group(1)}" if m else None

async def main():
    e = (await Runner.run(extractor, MAIL)).final_output
    for r in e.rows:
        print(normalize_sku(r.raw_sku), r.quantity, repr(r.quantity_text), r.urgency)
    vague = [r.raw_sku for r in e.rows if r.quantity is None]
    if e.uncertain_fields or vague:
        print("human review queue:", e.uncertain_fields, vague)
    else:
        print("passes automatically")

asyncio.run(main())

# Output:
# SKU-11 12 '12 pcs' unspecified
# SKU-42 None 'about ten' standard
# human review queue: ['quantity of the second row'] ['42']

Be careful about what this listing proves: the model's answer is, once again, the script I wrote (two rows, the second with a null quantity and the wording "about ten"). It does not say that a real model would extract it like this. What it does prove is that the schema with int | None is accepted by the SDK, that the object comes back typed, and that the downstream code can route the case: since one quantity is null and uncertain_fields is not empty, the mail ends up in human review. The function normalize_sku is deterministic; on the edge cases:

for attempt in ("SKU 11", "sku-11", "11", "0042", "code 11", "SKU-11-grey"):
    print(f"{attempt!r:16} -> {normalize_sku(attempt)}")

# Output:
# 'SKU 11'         -> SKU-11
# 'sku-11'         -> SKU-11
# '11'             -> SKU-11
# '0042'           -> SKU-42
# 'code 11'        -> None
# 'SKU-11-grey'    -> None

Four design choices, each with its reason.

• reasoning is the first field of Row. If the quote comes first, the model looks for it in the text before deciding the values; moved to the end it becomes an after-the-fact justification.
• quantity: int | None plus quantity_text. Without the None, "about ten" becomes 10 and you would never know it was the customer's approximation. It is the general pattern for asking a model not to make things up: give it a place to put the uncertainty.
• raw_sku instead of a normalised sku. Normalisation is a five-line deterministic function (the one above); asking the model for it means paying tokens for a probabilistic result. The model extracts, your code normalises.
• uncertain_fields. It is the channel that lets doubts surface: an extraction with that list non-empty goes into a review queue, not into automatic processing.

The four choices have a single logic, repeated four times: give the model a place to put what it does not know, and leave to the code everything that can be decided with a rule. The price is a wider schema, with fields that look redundant; the gain is that every doubt becomes a piece of data, not a false figure.

The field that makes an answer verifiable

There is an idea in this lesson that returns in all the following ones: a schema can force the model to declare what it relies on. It does not prevent the lie, but it makes it comparable, and therefore detectable by code. There are three forms, in order of strength.

The weakest: the list of sources used. A data_used: list[str] with the names of the tools consulted. The model could write anything in it, but you have the real record of the calls in the run, new_items, and you can compare the two lists. Here the script declares read_prices, a tool that does not exist and was never called:

from agents import ToolCallItem

def traceability_consistent(result) -> dict:
    """Compare what the model SAYS it used with what it actually used."""
    declared = set(result.final_output.data_used)
    actual = {i.raw_item.name for i in result.new_items if isinstance(i, ToolCallItem)}
    return {"consistent": declared == actual,
            "declared_never_called": sorted(declared - actual),   # the model made it up
            "called_not_declared": sorted(actual - declared)}     # it left it out

async def demo():
    result = await Runner.run(agent, "Should SKU-11 be reordered?")
    print(traceability_consistent(result))

asyncio.run(demo())

# Output:
# {'consistent': False, 'declared_never_called': ['read_prices'], 'called_not_declared': []}

The function reads the name only from the items of type ToolCallItem. A tool "declared but never called" is the signature of an answer built from memory and then dressed up with credibility; the opposite case, "called but not declared", is an omission, less serious. The signal costs one field in the schema and six lines of comparison.

Stronger: the literal quote. A quote: str field with the obligation to report the exact portion of the source text. The check is mechanical: the string is in the document or it is not.

def quote_is_valid(source_text: str, quote: str, min_length: int = 3) -> bool:
    """The quote must appear in the source, ignoring whitespace and case."""
    def norm(s: str) -> str:
        return " ".join(s.lower().split())
    if len(quote.strip()) < min_length:
        return False                          # empty or useless quote
    return norm(quote) in norm(source_text)

print(quote_is_valid(MAIL, "no rush for   the 42s"))
print(quote_is_valid(MAIL, "delivery by Friday"))
print(quote_is_valid(MAIL, "ok"))

# Output:
# True
# False
# False

The three outputs are read in order: the first quote is there (whitespace and case are normalised), the second does not appear in the mail, the third is too short to be meaningful. This function turns "did the model hallucinate?" from a subjective judgment into a boolean check, without a second model. It has a limit: a true quote attached to a wrong conclusion gets through.

The strongest: the reference to an entity of your system. An order_number: str field is not checked by similarity but with a query: if the number does not exist or does not belong to the authenticated user, the answer is false and you know it. There is no listing here because it depends on your database. The general rule is the same as for the other two: every time you can replace a judgment with a comparison, do it.

Where the real gain lies: these fields do not improve the answer, they make it controllable. If, as a hypothetical example, out of a hundred outputs four failed the automatic checks, a human would review four and not a hundred. The number is made up for the example; the mechanism, that human review shifts from the whole to the exception, is the point.

Evaluating an extractor (why "it seems to work" is not enough)

A valid schema says nothing about correctness. To know, you need a set of cases annotated by hand (twenty or thirty mails, with the right extraction written by you) and three measures that behave differently. In the listing every non-conforming case counts as entirely wrong also for per-field accuracy, otherwise the worst cases would vanish from the denominator and the measure would come out inflated.

from agents.exceptions import ModelBehaviorError

async def evaluate(agent, cases: list[tuple[str, dict]]) -> dict:
    """cases = [(text, expected_extraction_as_dict), ...]"""
    conforming = exact = fields_right = fields_total = 0
    for text, expected in cases:
        fields_total += len(expected)                  # a non-conforming case counts as entirely wrong
        try:
            got = (await Runner.run(agent, text)).final_output.model_dump()
        except ModelBehaviorError:
            continue                                   # non-conforming: no field right
        conforming += 1
        exact += int(got == expected)
        fields_right += sum(got.get(k) == v for k, v in expected.items())
    return {"conformity": round(conforming / len(cases), 3),       # does the schema hold?
            "exact_match": round(exact / len(cases), 3),           # everything right?
            "per_field_accuracy": round(fields_right / max(fields_total, 1), 3)}

EXPECTED = {"sku": "SKU-11", "outcome": "do_not_reorder", "quantity": 0,
            "reason": "4 in stock but 20 incoming",
            "data_used": ["read_stock", "read_incoming_orders"]}
test_cases = [("Should SKU-11 be reordered?", EXPECTED)] * 3

print(asyncio.run(evaluate(agent, test_cases)))

# Output:
# {'conformity': 0.667, 'exact_match': 0.333, 'per_field_accuracy': 0.4}

The three cases of the test are all the same question, with three answers I wrote: one exact, one conforming but with four fields out of five wrong, one non-conforming ("I don't know"). So conformity is 2/3 (0.667), exact match 1/3 (0.333), and per-field accuracy 6/15 (0.4): five right fields in the first case, one in the second, none in the third. Conformity measures the mechanism: if it is below 1 with constrained decoding something is wrong in the schema, if it is low without decoding you need the recovery path. Exact match is severe and realistic, because one wrong field out of eight makes the extraction wrong if the result goes through automatically. Per-field accuracy tells you where to intervene: discovering (hypothetical example) that urgency is right in 96% of cases and the address in 71% directs the work, whereas a single score hides it. Those thirty annotated cases are the most boring and most profitable part: without them, every change to the schema or the prompt is a bet.

The myth to debunk

The schema guarantees the shape, not the truth. A Decision(outcome="reorder", quantity=250, reason="stock below threshold") object is perfectly valid even when the stock is 1200 pieces. Constrained decoding and validation eliminate parsing errors, not hallucinations: they have made the data processable, not correct. The substantive check remains yours, and is done by comparing the fields with the source: if data_used is empty, the decision rests on nothing.

When NOT to use structured output

There are four cases, recognisable by precise symptoms.

When the recipient is a human being. If the answer is to be read by a person, prose is the right format. Forcing into a schema a text that you then recompose into sentences with a template means paying twice. The rule: structure what is processed by code, prose what is read by people; if you need both, two fields, one structured for the system and one of text for the user.

When you do not yet know what shape the data has. In the first explorations of an unknown corpus, the schema you would write today is probably wrong: you find out after looking at fifty examples. It is better to ask the model for free descriptions, collect them and derive the schema from those; starting from the schema means projecting your own expectations, and you will never see the fields you had not foreseen.

When the task is inherently open. "Write three ideas for this article": variety is the value. An ideas: list[str] works, but twelve mandatory fields turn a creative job into a form.

When the model does not hold the schema and you have no safety net. If you declare a type and do not implement the recovery path seen above, your code will fail in production on a fraction of requests, apparently at random: ModelBehaviorError without a reason. It is better to ask for prose and parse it tolerantly than to ask for a schema you cannot enforce.

The schema is a contract: versions and compatibility

The output type of an agent is an interface between two pieces of software, and like any interface it changes over time. Unlike an ordinary dataclass, though, there is a third party that has to "understand" the contract and has no version: the model. Three rules avoid most of the trouble.

Adding fields is compatible, removing and renaming is not. Renaming outcome to decision breaks every consumer and silently invalidates the annotated test cases. Beware, though, of a consequence of the strict schema seen above: a new field with a default ends up in required anyway, so the model must emit it; the default only helps your code when you build the object by hand.

Widening an enum is risky in an asymmetric way. If you add "postponed" to a Literal that had three values, the model will start using it, which is what you wanted, but the downstream code that did if outcome == "reorder": ... else: do_not_reorder() will send the postponed cases to "do not reorder" as well. The defence is an exhaustive match with a final branch that raises instead of assuming a default. To simulate the widened schema I use model_construct, which creates the object without validating it.

review_queue: list[Decision] = []

def apply(d: Decision) -> None:
    match d.outcome:
        case "reorder":
            create_proposal(d.sku, d.quantity)
        case "do_not_reorder":
            pass
        case "insufficient_data":
            review_queue.append(d)
        case other:
            # a new value in the schema arrives HERE, not in the wrong branch
            raise ValueError(f"unhandled outcome: {other!r} - update apply()")

base = dict(sku="SKU-11", quantity=250, reason="below threshold", data_used=[])
apply(Decision(outcome="reorder", **base))
apply(Decision(outcome="insufficient_data", **base))
print("in review:", len(review_queue))
try:
    # simulate a widened schema: model_construct skips validation
    apply(Decision.model_construct(outcome="postponed", **base))
except ValueError as e:
    print("ValueError:", e)

# Output:
# order proposal: 250 x SKU-11
# in review: 1
# ValueError: unhandled outcome: 'postponed' - update apply()

In the output you see the three paths: "reorder" creates the proposal, "insufficient_data" ends up in the review queue, and the new value "postponed" does not fall silently into a wrong branch but reaches the line that raises. In production, you see that error on the first day and not in the third month.

Version the schema together with the test cases. Schema, prompt and set of annotated cases are a single unit: changing one invalidates the measurements made with the others. Keeping them in the same folder and in the same commit is what lets you say "this version is better than the previous one" with an honest comparison.

An agent with a declared output type is a component, not a product: it is the property that makes it usable as a sub-agent in an orchestration (09-09), as a classifier inside a guardrail (09-10) and as a testable unit in a suite.

Trade-offs

• Structure versus flexibility. A closed type gives you control and cuts out the answers you had not foreseen, including correct ones. Mitigation: a note: str field where the model can add what the schema does not provide for.
• Quality versus constraint. Constraining the format can worsen the content, because the model loses the space in which it reasoned by writing. Remedy: a reasoning: str field before the decision fields.
• Portability. Constrained decoding is a feature of the server, not of the SDK. The same code on another server or another model may fall back on the instruction in the prompt: you need the recovery path, otherwise it works in development and nowhere else.
• Cost of the retry. Every retry is one more whole run, with its tokens and its latency.

The four items are the same tension seen from different sides: every guarantee you add (closed type, constraint, validation, retry) is paid for in model freedom, in latency or in recovery code. For a component that feeds another program the price is worth it; for an answer a person reads, usually not.

The traps people really fall into

1. Catching the wrong exception. With output_type the SDK raises ModelBehaviorError, without the offending field; ValidationError never reaches your except.
2. Mixing tools and structured output without planning for the error. If a tool fails and the schema has no value for "I could not decide", the model fills the fields with something. The "I don't know" state is mandatory.
3. Everything nullable. If every field is X | None, the model can return an object of only nulls, perfectly valid. Make non-nullable what you really need.
4. Putting the reasoning after the decision. A reason field after outcome is an after-the-fact justification of a choice already made.
5. Types the strict schema does not express well. dict[str, int] is rejected, Decimal becomes number-or-string: use str with a declared format ("ISO 8601 date, e.g. 2026-10-09") and convert yourself.
6. Believing that a valid object is a true object. It is the main trap of this lesson, and it is the reason the traceability field exists.

The order here does not matter, unlike the steps of a procedure: they are six independent ways of going wrong. The first two are found at the first test, the third and the fourth stay silent until someone looks at the data, and the last two are the ones that reach production.

What you can do now

You can declare the type of an agent's output and read the schema the SDK derives from it; you know what your code sees when the answer is defective and how to obtain the precise reason for a retry; you can design a schema that leaves the model a place for uncertainty, and add a field that makes the answer comparable with reality. What stays out is what I could not test here: how a real model, on Ollama, fills and respects a schema. In the next lesson we tackle the part that makes a conversation a conversation: memory, and its cost.

Sources

• OpenAI Agents SDK: Agents - the output_type parameter, the accepted types (dataclasses and TypedDict too), and the fact that it enables structured outputs, consulted on 2026-11-12
• OpenAI Agents SDK: Agent output - AgentOutputSchema, validate_json which must raise ModelBehaviorError if the JSON is not valid, consulted on 2026-11-12; the message without detail, the null run_data and the cases in the table are my own observations with 0.24.0
• Ollama: Structured outputs - enforcing a JSON schema on the answers and use through response_format in the compatible API, consulted on 2026-11-12
• Ollama: OpenAI compatibility - response_format among the supported fields of /v1/chat/completions, consulted on 2026-11-12
• Pydantic: JSON - model_validate_json and validate_json with strict validation, consulted on 2026-11-12
• Pydantic: JSON Schema - how types become schema (Literal into enum, ge/le into minimum/maximum), consulted on 2026-11-12
• JSON Schema 2020-12: Validation - the keywords minimum, maximum, maxLength mentioned in the schema, consulted on 2026-11-12
• Python: the match statement - the case with final capture used in apply, consulted on 2026-11-12

Run twice with identical output (Python 3.14.4, openai-agents 0.24.0, pydantic 2.14.3) against a fake model with a simulated HTTP transport: schema, validation, eight defective answers, tolerant schema and retry, extraction from the mail, traceability, quote, evaluation, exhaustive match, types in the schema. Not run: the token-by-token illustration (conceptual), the field with a reference to an entity of your system, and any call to a real model (not even Ollama). The percentage examples (4 out of 100, 96% and 71%) are hypothetical.