84 - Your First Working Agent: the Same Code on a Managed SDK and on Ollama

A minimal agent is three objects: a function decorated as a tool, an Agent with instructions and a list of tools, a Runner that executes the loop. Nothing else. In this lesson we write it with the OpenAI Agents SDK — the Python package openai-agents, which is the teaching subject of the module — and then, without touching the logic, we run it on a local model served by Ollama, changing only the client. It is the same exercise twice, and the comparison teaches more than either version taken alone.

Version used and what I ran. The code of this lesson is written against openai-agents 0.24.0 (installed without a version constraint in a temporary virtual environment on 2026-11-12) with Python 3.14.4. This SDK is young and its public surface changes often: pin the version in your requirements.txt and read the changelog before upgrading. I called no model: the blocks with output reported run against a "fake Ollama" (a simulated HTTP transport, no network, no port) described below. The blocks that talk to a real Ollama, to a managed endpoint, or that use streaming are patterns to adapt, not executed, and say so.

The three objects and who talks to whom

Before the code, the map. There are three objects of yours (the tool, the agent, the runner) and two things outside your process (the model and the system the tool contacts). The arrow that counts is the dashed one: the model never reaches your system. It asks, and the runner executes.

Installation and the "hello world"

python -m venv .venv && source .venv/bin/activate
pip install "openai-agents==0.24.*"

The constraint ==0.24.* accepts any patch of 0.24 and rejects 0.25: it is the right level of strictness for a package that changes between minors. The pip command was not run with this constraint (I installed without a constraint and got 0.24.0). The first agent, the one that does nothing agentic, is this (pattern, not executed: without a configured model the SDK would default to the one of the OpenAI API, which requires a key).

import asyncio
from agents import Agent, Runner

agent = Agent(
    name="Warehouse assistant",
    instructions=(
        "Answer in English, tersely. "
        "Do not invent numbers: if you lack a datum, say so."
    ),
)

async def main():
    result = await Runner.run(agent, "Explain in two lines what a reorder threshold is.")
    print(result.final_output)

asyncio.run(main())

Three observations that are not details. The first: Runner.run is a coroutine, which is why the previous lesson talked about asyncio; there is also Runner.run_sync for scripts, but it must not be used inside code that is already asynchronous (a server, a notebook with an active loop), where it would raise a loop-already-running error. The second: the return value is not a string but a result object; final_output is the answer, but there is much more in there, and we come back to it shortly. The third: there is no tool, so this is not yet an agent, it is a call to a model with a wrapper, and the loop ends at the first turn.

A fake Ollama, to try the SDK without a model

To really run the code that follows without a model I built a simulated HTTP transport: AsyncOpenAI accepts a custom http_client, and I pass it a client whose transport answers with ready-made "chat completions" responses, one per request, taken from a script. This runs the real SDK (schema generation, loop, turn ceiling, result structure) and records every request body in REQUESTS, so I can check what the SDK would send. It verifies nothing about the behaviour of a real model: the "intelligent" answers are written by me. In my test environment the HTTP package was called httpx2 (I import it as httpx); in yours, with a normal installation, it will be httpx.

import asyncio, json
import httpx2 as httpx
from openai import AsyncOpenAI
from agents import (Agent, Runner, function_tool, ModelSettings,
                    OpenAIChatCompletionsModel, set_tracing_disabled)

set_tracing_disabled(True)

# --- a fake "Ollama": no network, no port ---------------------------
REQUESTS = []          # copy of every request body received

def tool_reply(n, name, args):
    return {"id": f"r{n}", "object": "chat.completion", "created": 0, "model": "fake",
            "choices": [{"index": 0, "finish_reason": "tool_calls", "message": {
                "role": "assistant", "content": None,
                "tool_calls": [{"id": f"call_{n}", "type": "function",
                                "function": {"name": name, "arguments": json.dumps(args)}}]}}],
            "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}}

def text_reply(n, text):
    return {"id": f"r{n}", "object": "chat.completion", "created": 0, "model": "fake",
            "choices": [{"index": 0, "finish_reason": "stop",
                         "message": {"role": "assistant", "content": text}}],
            "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}}

def script_factory(script):
    """script: list of JSON replies, one per request."""
    counter = {"n": 0}
    def handler(request: httpx.Request) -> httpx.Response:
        REQUESTS.append(json.loads(request.content))
        r = script[min(counter["n"], len(script) - 1)]
        counter["n"] += 1
        return httpx.Response(200, json=r)
    return handler

def fake_client(script):
    http = httpx.AsyncClient(transport=httpx.MockTransport(script_factory(script)))
    return AsyncOpenAI(base_url="http://localhost:11434/v1", api_key="ollama", http_client=http)

The script is a list of replies: tool_reply is an assistant message with a tool request, text_reply is the final answer. With min(counter["n"], len(script) - 1) the last reply repeats if the model is queried more times than expected, which is useful for trying the turn ceiling.

Adding a tool: the moment it becomes an agent

The @function_tool decorator does three things at once: it exposes the function to the model, it derives the JSON schema of the arguments from the type hints, and it uses the docstring as the description. This means the docstring is not a comment: it is part of the prompt, and the model reads it to decide whether to call you. The warehouse, the two tools and the instructions are those of the previous lesson; the tools return an error dictionary for a nonexistent SKU, instead of raising. The block goes at the end of the one above (same file).

WAREHOUSE = {
    "SKU-11": {"stock": 4, "threshold": 10, "incoming": 20},
    "SKU-42": {"stock": 120, "threshold": 30, "incoming": 0},
}

@function_tool
def read_stock(sku: str) -> dict:
    """Current stock and reorder threshold of a product code (SKU).

    Use this tool every time you need to know a quantity in the warehouse.
    Never estimate the stock from memory.
    """
    row = WAREHOUSE.get(sku)
    if row is None:
        return {"error": "unknown_sku", "sku": sku,
                "known": sorted(WAREHOUSE)}          # a useful error, not an exception
    return {"sku": sku, "stock": row["stock"], "threshold": row["threshold"]}

@function_tool
def read_incoming_orders(sku: str) -> dict:
    """Quantity already ordered and in transit for a SKU, not yet in stock."""
    row = WAREHOUSE.get(sku)
    if row is None:
        return {"error": "unknown_sku", "sku": sku}
    return {"sku": sku, "incoming": row["incoming"]}

INSTRUCTIONS = ("Decide whether a product should be reordered. "
                "Always read the data with the tools, never from memory. "
                "Take into account what is already incoming. "
                "Finish with one line: REORDER <quantity> or DO NOT REORDER, plus the reason.")

SCRIPT = [tool_reply(1, "read_stock", {"sku": "SKU-11"}),
          tool_reply(2, "read_incoming_orders", {"sku": "SKU-11"}),
          text_reply(3, "DO NOT REORDER: 4 in stock but 20 incoming.")]

def create_agent(script, **extra):
    model = OpenAIChatCompletionsModel(model="llama3.1:8b", openai_client=fake_client(script))
    return Agent(name="Warehouse assistant", instructions=INSTRUCTIONS,
                 tools=[read_stock, read_incoming_orders], model=model, **extra)

async def main():
    agent = create_agent(SCRIPT)
    result = await Runner.run(agent, "Should SKU-11 be reordered?", max_turns=6)
    print(result.final_output)
    print(len(REQUESTS), "requests to the model")
    print(json.dumps(REQUESTS[0]["tools"][0], ensure_ascii=False, indent=1))

asyncio.run(main())

# DO NOT REORDER: 4 in stock but 20 incoming.
# 3 requests to the model
# {
#  "type": "function",
#  "function": {
#   "name": "read_stock",
#   "description": "Current stock and reorder threshold of a product code (SKU).\n\nUse this tool every time you need to know a quantity in the warehouse.\nNever estimate the stock from memory.",
#   "parameters": {
#    "properties": {
#     "sku": {
#      "title": "Sku",
#      "type": "string"
#     }
#    },
#    "required": [
#     "sku"
#    ],
#    "title": "read_stock_args",
#    "type": "object",
#    "additionalProperties": false
#   },
#   "strict": true
#  }
# }

The output shows three things. The (fake) model received three requests: two for the tools, one for the answer, so three turns, and the path was not decided by your code. The schema sent for read_stock has the name of the function, the whole docstring as the description, and sku as a required string derived from the type hint; there is also "strict": true with additionalProperties: false, which the SDK adds by default (it holds for the protocol I tried: check what your endpoint accepts). If you remove the type hint or the docstring, the schema gets worse without warning you.

Note max_turns=6: it is the safety ceiling we talked about. If it is exceeded, the SDK raises MaxTurnsExceeded, importable from agents.exceptions: catch it, because it is an expected case. I try it further down.

What is inside the result

This is where those who make demos are separated from those who make systems. The object returned by Runner.run contains at least four things that count. final_output is the final answer (a string, or the typed object if you declared an output_type, as in Structured output). new_items is the list of everything that was generated in the run (messages, tool requests, tool results, handoffs): it is the record of what really happened, and the only honest way of checking whether an action was executed or only declared. last_agent says which agent produced the answer, information that is needed only with handoffs (Handoffs). to_input_list() returns the history ready to be reused as the input of the next turn, and it is the building block of memory (Conversational memory).

Get into the habit of printing new_items the first time you write a new agent. Almost all surprises ("why did it answer that?") are explained by looking at which tools it called and with which arguments. A print function written once and kept forever becomes your primary diagnostic tool, the one you use before any hypothesis. The class names are those of my environment, in agents.items: they are among the things that move, and if the import does not resolve you can fall back on type(item).__name__ comparing strings.

import asyncio
from agents.items import ToolCallItem, ToolCallOutputItem, MessageOutputItem
from base_lib import *      # the fake Ollama and the tools of the example above, in one file
def print_run(result) -> None:
    """Print in order everything that really happened in the run."""
    for i, item in enumerate(result.new_items, 1):
        kind = type(item).__name__
        if isinstance(item, ToolCallItem):
            print(f"{i:2d}. TOOL REQUESTED  {item.raw_item.name} {item.raw_item.arguments}")
        elif isinstance(item, ToolCallOutputItem):
            print(f"{i:2d}. TOOL RESULT     {str(item.output)[:160]}")
        elif isinstance(item, MessageOutputItem):
            print(f"{i:2d}. MESSAGE         {str(item.raw_item.content[0].text)[:160]}")
        else:
            print(f"{i:2d}. {kind}")
    print("-" * 60)
    print("final agent:", result.last_agent.name)
    print("answer     :", result.final_output)

async def main():
    result = await Runner.run(create_agent(SCRIPT), "Should SKU-11 be reordered?")
    print_run(result)

asyncio.run(main())

#  1. TOOL REQUESTED  read_stock {"sku": "SKU-11"}
#  2. TOOL RESULT     {'sku': 'SKU-11', 'stock': 4, 'threshold': 10}
#  3. TOOL REQUESTED  read_incoming_orders {"sku": "SKU-11"}
#  4. TOOL RESULT     {'sku': 'SKU-11', 'incoming': 20}
#  5. MESSAGE         DO NOT REORDER: 4 in stock but 20 incoming.
# ------------------------------------------------------------
# final agent: Warehouse assistant
# answer     : DO NOT REORDER: 4 in stock but 20 incoming.

Here everything can be read: two tool turns, the right ones, in the right order, with the correct arguments. (This is the script's run, so it demonstrates the print function, not the reasoning of a model.) If with a real model the final answer were "REORDER 6" with these same lines above it, you would know the problem is in the final reasoning and not in the data collection, and the cure would be in the instructions, not in the catalogue. If instead there were no "TOOL REQUESTED" line, you would know the model answered from memory, which is a completely different problem.

Two other cases worth trying right away, still with the script: a model that never stops asking for tools, and a tool that returns an error instead of raising.

from agents.exceptions import MaxTurnsExceeded

async def main():
    loop = [tool_reply(i, "read_stock", {"sku": "SKU-11"}) for i in range(1, 10)]
    try:
        await Runner.run(create_agent(loop), "Spinning in circles", max_turns=3)
    except MaxTurnsExceeded as e:
        print("MaxTurnsExceeded:", e)

    c = [tool_reply(1, "read_stock", {"sku": "SKU-99"}),
         text_reply(2, "SKU-99 does not exist. Known: SKU-11, SKU-42.")]
    result = await Runner.run(create_agent(c), "What about SKU-99?")
    for item in result.new_items:
        if type(item).__name__ == "ToolCallOutputItem":
            print(item.output)

asyncio.run(main())

# MaxTurnsExceeded: Max turns (3) exceeded
# {'error': 'unknown_sku', 'sku': 'SKU-99', 'known': ['SKU-11', 'SKU-42']}

The first line is the ceiling in action: with three turns granted and a model that always asks for the same tool, the SDK raises the dedicated exception instead of running forever. The second is the reason it pays to return an error dictionary: the nonexistent SKU did not stop the run, it went back to the model with the list of known SKUs, and the model (here, the script) can correct itself.

What the SDK does that you do not see (and that is not worth rewriting)

In the lesson on anatomy we wrote the loop by hand, and the conclusion was that there is no magic. It is true, and it is not a reason for not using an SDK: between the teaching loop and something that survives contact with reality there are five layers of boring work, and they are exactly the layers you do not want to write twice.

Schema generation. Deriving a correct JSON schema from the signature of a Python function looks easy and is not: optional types, unions, nested models, lists of objects, default values, field aliases, and the subset of JSON Schema that each endpoint accepts — which is not the same for all. The SDK does this work, and redoes it when the endpoint changes the rules.

Protocol normalisation. There are at least two families of APIs for talking to a model, with different shapes for messages, tools and responses, and each has dialects. An SDK gives you a single interface; the price is that sometimes a feature exists on one protocol and not on the other — like the Ollama case we see shortly.

Handling parallelism on tools. The model can ask for three tools in one answer. Executing them, collecting their results in the right order, pairing each with its own correlation identifier, and handling the case where one of the three raises while the other two succeed: it is the code that in that lesson I deliberately simplified, and in production it cannot be simplified.

Counting turns and tokens, and the structure of the result. It is not hard, it is easy to forget and tedious to keep consistent across the various exit points of the loop.

Extension points. Guardrails, lifecycle hooks, tracing, memory sessions: they are all things you can write, and that if you write them from scratch you end up intertwining with the business logic. Having them as declarative hooks keeps the logic clean (guardrails and tracing come later in the module).

The flip side remains, which concerns a decision to take consciously: a young SDK is a dependency that moves. Class names that change position, parameters renamed, default behaviours that flip between one minor and the next.

Three precautions make the risk manageable. The first: pin the version with a strict constraint in the dependencies file, and upgrade on purpose by reading the changelog, not out of the inertia of a pip install -U. The second: do not scatter the SDK's imports all over the project; a thin module that exposes your functions (create_agent, execute) and imports the SDK only there turns a broken upgrade from a hunt across forty files into a change in one. The third: keep the tools independent of the SDK, writing read_stock_impl(sku) as a normal Python function and applying the decorator to a thin wrapper, so that tests run on the pure function and do not break when the decorator changes signature.

This third point is also the reason why writing the loop by hand is not wasted time: when the abstraction behaves unexpectedly, you will know what is underneath, and the difference between "I don't understand why" and "ah, it isn't appending the assistant's message" is an afternoon.

Ollama in practice: choosing the model and knowing what to expect

The line of code that points the agent at Ollama is a single one (you see it further down), and it hides three decisions that are worth taking with awareness.

The first: the model must support tools. It is a property of the model, not of the server. A model trained without examples of function calling will not emit structured tool requests: it will ignore the catalogue and answer in prose, and the symptom will be identical to that of a badly described tool. Before spending an afternoon rewriting docstrings, check on the model's card in the Ollama library whether tool calling is among its declared capabilities.

The second: the size of the model determines the quality of tool calling more than any prompt. A model of a few billion parameters tends to handle two or three tools with sharp boundaries better than ten (it is a common observation, not a measurement of mine). The design consequence is concrete: on a small local model, the catalogue must be kept shorter and the names more explicit. It is not a defect to fix, it is a constraint to respect — and it often produces a better architecture, because it forces you to split an omniscient agent into specialists.

The third: the effective context window may not be the declared one. Local servers can apply a default context value lower than the one the model would support, so as not to run out of memory; check the value in the documentation of your version of Ollama. It is one of the most frustrating causes of inexplicable behaviour: the agent "forgets" the beginning of the conversation at the fourth turn not because the model is poor, but because the context was silently truncated by the server. It is worth checking the context parameter of your configuration before blaming the model.

On what changes in the results, two honest observations. The first: with few tools, precise instructions and typed output, the gap between a mid-sized local model and a frontier one is expected to narrow, because much of the work is done by the format constraint and the schema. The second: with many steps, many similar tools or contradictory data, the gap is expected to widen. These are expectations, not results of mine. The practical conclusion is not "one is better than the other" but: measure on your task, with your tools, on twenty real cases. It is half a day of work and it replaces months of opinions.

Streaming: watching the agent while it thinks

With Runner.run the user looks at an empty screen for the whole duration of the run, which with several turns can be tens of seconds. Runner.run_streamed returns an object you iterate over as events arrive. The block is a pattern to adapt, not executed: streaming requires a streamed response that my simulated transport does not produce. I only checked that Runner.run_streamed and ResponseTextDeltaEvent exist in 0.24.0.

import asyncio
from agents import Runner
from openai.types.responses import ResponseTextDeltaEvent

async def main():
    stream = Runner.run_streamed(agent, "Should SKU-11 be reordered?")
    async for event in stream.stream_events():
        if event.type == "raw_response_event" and isinstance(event.data, ResponseTextDeltaEvent):
            print(event.data.delta, end="", flush=True)      # text, token by token
        elif event.type == "run_item_stream_event":
            name = type(event.item).__name__
            if name == "ToolCallItem":
                print(f"\n[consulting: {event.item.raw_item}]")
    print("\n\nfinal:", stream.final_output)      # available once the stream is exhausted

asyncio.run(main())

Two things to know, both to be checked in the documentation on streaming. The first: final_output on the streaming object is reliable only after you have consumed all the events — before that it may not be there yet. The second, less obvious: streaming does not reduce the total latency, it reduces the perceived latency. The work is the same; what changes is that the user sees something happening. On a multi-turn agent the move that really improves the experience is to announce the steps ("I am consulting the warehouse"), not to make characters scroll.

Model parameters: what is worth touching

The model configuration object, ModelSettings, exposes the decoding parameters. Three count for an agent, the others almost never. I checked what the SDK really sends by reading the request body in the fake Ollama (the block uses await at the top level: inside an async function, as in the earlier examples).

from agents import Agent, ModelSettings

ag = create_agent(SCRIPT, model_settings=ModelSettings(
    temperature=0.0,            # repeatable decisions: no creativity
    tool_choice="required",     # on the first turn it MUST use a tool
    parallel_tool_calls=True,   # several tools at once when they are independent
    max_tokens=600,
))
await Runner.run(ag, "Should SKU-11 be reordered?")
print([r.get("tool_choice") for r in REQUESTS])      # one per request to the model
print({k: REQUESTS[0].get(k) for k in ("temperature", "tool_choice", "parallel_tool_calls", "max_tokens")})

# ['required', None, None]
# {'temperature': 0.0, 'tool_choice': 'required', 'parallel_tool_calls': True, 'max_tokens': 600}

The first line is the discovery: tool_choice appears as required only in the first request, and is absent in the next two. With 0.24.0 the SDK resets the forced choice by itself after a tool use, otherwise the model could never close with a text message and you would hit the turn limit. I did not verify this for earlier versions; check the behaviour of yours. The second line confirms that the other three parameters go out exactly as you wrote them.

temperature=0 is almost always right for an agent that decides: creativity is for writing texts, not for choosing whether to call read_stock. Zero does not mean deterministic, though: on real infrastructure batching and floating-point arithmetic can leave residual variability, and it is one of the reasons why testing agents is a statistical problem, not an assertion problem. tool_choice="required" is the direct remedy for the agent that answers without looking at the data, with an important condition: Ollama's OpenAI-compatible endpoint lists tool_choice among the unsupported fields (compatibility page, consulted on 2026-11-12), so on Ollama you cannot count on this lever and you have to check yourself, in the loop, that the first turn called a tool. parallel_tool_calls cuts turns when the tools are independent (two reads on different SKUs) and should be turned off when they have side effects or depend on each other, because in parallel the order is not guaranteed.

The same agent on Ollama, locally

Ollama exposes an endpoint compatible with OpenAI's chat completions API at http://localhost:11434/v1 (documentation). This is enough to reuse the same SDK with a model running on your machine: no key, no data leaving, zero marginal cost. The block is a pattern to adapt, not executed: the only difference from the tests above is that the client points to a real server instead of the simulated transport.

import asyncio
from openai import AsyncOpenAI
from agents import (Agent, Runner, function_tool,
                    OpenAIChatCompletionsModel, set_tracing_disabled)

client = AsyncOpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
set_tracing_disabled(True)        # managed tracing would need a real key

local_model = OpenAIChatCompletionsModel(model="llama3.1:8b", openai_client=client)

agent = Agent(
    name="Warehouse assistant",
    instructions="...the same instructions as before...",
    tools=[read_stock, read_incoming_orders],
    model=local_model,          # the only line that really changes
)

asyncio.run(Runner.run(agent, "Should SKU-11 be reordered?"))

First you need to have the model: ollama pull llama3.1:8b (not executed). api_key="ollama" is a placeholder: the client demands it, the server ignores it. The fake Ollama of the previous blocks also uses the same placeholder, and it is exactly the construction you see above with a different transport.

What really changes, to know before finding out in production. Function calling is there but tends to be more fragile on an 8-billion-parameter model, which gets the choice of tool and the arguments wrong more often than a frontier one (a common observation, to be measured on your task); it is recovered with more explicit descriptions, fewer tools and instructions that say when to use what. Not all local models support tools, and it is a property of the model, not of Ollama. Managed tracing is not there and has to be replaced with a tracing of your own, as we will see later in the module. The features tied to the "responses" API are not available: the compatible endpoint is the chat completions one, and the models chapter of the SDK describes how the SDK tells the two protocols apart. Finally, latency depends on your hardware: on CPU an 8B model can be slow, the difference with a GPU is large.

Making the model a parameter

All the previous reasoning has a single architectural consequence, and it is best applied on day one because afterwards it costs: the model must not appear in the code that defines the agents. If the model object comes from a configuration function, trying a different provider, comparing two models on the same task or running the tests without spending becomes a change of environment variable. In the first version of this example the "managed" branch built a dictionary with both environment variables and failed with KeyError if one was missing, even when only one was needed; I found out by running the block and fixed it by reading only the variable of the requested profile.

import os
from functools import lru_cache
from openai import AsyncOpenAI
from agents import Agent, OpenAIChatCompletionsModel, set_tracing_disabled, function_tool

@function_tool
def read_stock(sku: str) -> dict:
    """Stock of a SKU."""
    return {}

@lru_cache(maxsize=None)          # one client per process, not one per call
def model(profile: str = "default"):
    """Return the model object according to the environment configuration.

    profile: "default" for the work, "small" for guardrails and classifiers.
    """
    backend = os.getenv("AI_BACKEND", "ollama")

    if backend == "ollama":
        client = AsyncOpenAI(base_url=os.getenv("OLLAMA_URL", "http://localhost:11434/v1"),
                             api_key="ollama")                  # placeholder, ignored
        set_tracing_disabled(True)                              # no managed tracing
        names = {"default": os.getenv("OLLAMA_MODEL", "llama3.1:8b"),
                 "small": os.getenv("OLLAMA_MODEL_SMALL", "llama3.2:3b")}
        return OpenAIChatCompletionsModel(model=names[profile], openai_client=client)

    if backend == "managed":
        variable = {"default": "MODEL_DEFAULT", "small": "MODEL_SMALL"}[profile]
        return os.environ[variable]    # the SDK also accepts the name as a string

    raise ValueError(f"AI_BACKEND not recognised: {backend!r}")

WAREHOUSE_INSTRUCTIONS = "Decide whether to reorder."      # in reality: from a versioned file, not inline

def create_warehouse_agent():
    return Agent(name="Warehouse assistant", instructions=WAREHOUSE_INSTRUCTIONS,
                 tools=[read_stock], model=model("default"))

a = create_warehouse_agent()
print(type(a.model).__name__, a.model.model)
print(model("default") is model("default"))
print(type(model("small")).__name__, model("small").model)
os.environ["AI_BACKEND"] = "managed"; os.environ["MODEL_DEFAULT"] = "model-name"
model.cache_clear()
print(model("default"))
os.environ["AI_BACKEND"] = "unknown"; model.cache_clear()
try:
    model("default")
except ValueError as e:
    print("ValueError:", e)

# OpenAIChatCompletionsModel llama3.1:8b
# True
# OpenAIChatCompletionsModel llama3.2:3b
# model-name
# ValueError: AI_BACKEND not recognised: 'unknown'

The output verifies four behaviours. The Ollama branch builds an OpenAIChatCompletionsModel with the default name; the cache returns the same object on the second call (True), so one client per process; the "small" profile selects another name; the "managed" branch returns the name as a string; an unknown backend fails immediately with a clear message. tool, WAREHOUSE_INSTRUCTIONS and the rest are reduced to the minimum to make the block run: in the real code the tools are the ones above and the instructions live in a file.

Four things this function does that are worth copying. The lru_cache guarantees one client per process: without it every call creates a new one, throws away the connection pool and pays a pointless TLS handshake. The profile separates the "work" model from the small one for guardrails and classifiers, which run on every request and where an expensive model doubles the system's bill without improving anything. The model names from environment variables let you change size without touching the code, which is what you will want to do the first time you measure. And the explicit failure on an unrecognised backend avoids the worst of the outcomes: a wrong configuration that silently uses the default and makes you measure the wrong thing. A note on what not to put in that function: the instructions. The model changes, not the mandate; keeping them in a versioned file next to the code lets you know, from the git history, which prompt was active when that strange behaviour showed up.

Trade-off: managed SDK or local model?

Five axes to decide. On tool-calling quality the advantage generally goes to frontier models, especially on large tool catalogues and multi-step reasoning (to be measured, see below). On cost, the managed endpoint is paid per token and the bill grows with the history, the local one is paid in hardware and electricity with an almost null marginal cost per request, and the break-even point depends on volume and has to be calculated, not felt. On privacy and contractual constraints, with a local model the data does not leave the machine, and for sensitive documents it is often the only admissible option.

On reproducibility, a local model with a pinned tag is the same model six months from now, while a managed endpoint can update the model under you and your tests change result without your having touched anything. On operations, the local one you have to run, update, size and monitor yourself. The choice that always holds, whatever the verdict: write the code so that the model is a parameter, as in the previous section.

The first comparison to make: twenty cases, two models

Measuring on your own task replaces months of opinions, and it is simpler than it looks. You do not need an evaluation framework: you need twenty real requests, taken from the logs and not invented, and for each the decision you consider correct. Then you look at three numbers, and none of the three is "the quality of the answer". The block below is the skeleton, tried on the fake Ollama with only three cases and three answers written by me, one of them deliberately "from memory": it serves to check the measuring code, not to compare models.

import asyncio
from base_lib import *

async def compare(build_agent, cases) -> dict:
    right_tools = right_outcomes = total_turns = 0
    for request, expected_tools, expected_outcome in cases:
        result = await Runner.run(build_agent(request), request, max_turns=8)
        called = {i.raw_item.name for i in result.new_items
                  if type(i).__name__ == "ToolCallItem"}
        right_tools += int(expected_tools.issubset(called))       # did it look at the data?
        right_outcomes += int(result.final_output.startswith(expected_outcome))
        total_turns += len(result.raw_responses)                  # one model response per turn
    n = len(cases)
    return {"tool_coverage": right_tools / n,
            "correct_outcome": right_outcomes / n,
            "mean_turns": total_turns / n}

SCRIPTS = {
    "Should SKU-11 be reordered?": SCRIPT,
    "What about 42?": [tool_reply(1, "read_stock", {"sku": "SKU-42"}),
                       text_reply(2, "DO NOT REORDER: 120 in stock, threshold 30.")],
    "I need SKU-99": [text_reply(1, "REORDER 10: SKU-99 seems to be out.")],  # answers from memory
}
CASES = [("Should SKU-11 be reordered?", {"read_stock", "read_incoming_orders"}, "DO NOT REORDER"),
         ("What about 42?", {"read_stock"}, "DO NOT REORDER"),
         ("I need SKU-99", {"read_stock"}, "INSUFFICIENT DATA")]

async def main():
    print(await compare(lambda r: create_agent(SCRIPTS[r]), CASES))

asyncio.run(main())

# {'tool_coverage': 0.6666666666666666, 'correct_outcome': 0.6666666666666666, 'mean_turns': 2.0}

The result is read at once: the third case ("I need SKU-99") has an answer written on purpose without consulting the tools and with a wrong outcome, and this lowers coverage and outcome to 2/3; the mean turns are 2.0 (3 + 2 + 1 model responses, over 3 cases). With a real model the three numbers would be your measurements. Why these three. Tool coverage measures the thing that breaks first on small models: the discipline of consulting the data instead of answering from memory; if it is low, the problem is not the reasoning but adherence to the instructions, and you attack it with the catalogue and the instructions, not by changing model. The correct outcome is the number that counts for the user, but on its own it does not say where to intervene. The mean turns are the cost: two models with the same outcome but one at 2.3 turns and the other at 4.8 are not equivalent, because the second costs twice as much and answers twice as late (example numbers).

The combined reading gives useful diagnoses. High coverage and low outcome: the model looks at the data and reasons badly, and here a bigger size really helps. Low coverage and high outcome: the model guesses, and will guess until the day it does not; it has to be corrected even if the final score looks good. Mean turns that explode with the outcome unchanged: the model hesitates, usually because of an ambiguous catalogue or a vague completion criterion. One last precaution: every case must be run more than once. With temperature=0 the variability is small but not zero, and a comparison on a single pass can reverse on the second. Three passes over twenty cases are sixty runs, which on a local model cost only time.

The traps people really fall into

1. Forgetting the type hint or the docstring on a tool. Without a type hint the schema is not generated (or becomes permissive); without a docstring the model does not know when to call it and ignores it. The symptom is identical: "the tool is there but it never uses it".
2. Raising exceptions inside a tool for wrong inputs. A nonexistent SKU is not a bug: it is a situation the agent can correct, if you tell it about it. Return an error dictionary and the model tries again with the right argument.
3. Believing final_output about actions. "I updated the warehouse" is prose. The proof is in new_items and in your application log.
4. Blocking tools. A synchronous function that runs a slow query blocks the event loop. Declare it async def and use an async client, or wrap it with asyncio.to_thread.
5. Putting credentials in tool arguments. If a parameter is token: str, the model will try to invent it. Secrets are read from the environment inside the function; they must not appear in the schema.
6. Not pinning the SDK version. On a young package, an unplanned pip install -U is a proven way of breaking the build on a Friday evening.

The six traps divide into two families. The first three are errors of contract between you and the model (schema, errors, trust in prose): the program runs and does the wrong thing. The last three are errors of engineering (blocking the event loop, secrets in the schema, moving dependencies): the program works until you put it under load or upgrade it. Both are prevented with the same habits of this lesson: read new_items, keep the tools small and pure, pin the versions.

What you can do now

You can write an agent with three objects, tool, agent and runner; read what it really did with new_items; try the SDK without a model using a simulated transport; and move the same agent to a local model by changing one line, keeping the model as a configuration parameter. The SDK adds no intelligence, it adds ergonomics: automatic schema from type hints, loop handling, turn ceiling, result structure. In the next lesson, Tools: function calling, JSON schemas and errors, we go inside those tools: how the schema is born, how the arguments are validated, and why errors should be returned and not raised.