05 - AI Agents: ReAct, tool-use, planning

You have a model that can write an answer, but the answer you need depends on a piece of data the model does not have: how many copies of a book are free in a library today, what a file says, what a calculation gives with the right numbers. An agent is the simplest solution to this problem: a loop in which the model proposes an action, your program executes it and hands back the result, and the model starts again from that result until it decides to answer. The model never executes anything: it writes text, and everything that happens in the world is done by the program around it.

This is the fifth article of the series Modern Artificial Intelligence. It follows the previous one on advanced RAG, because searching a document base is one of the ways an agent consults the world (you will find the basics in the article on embeddings and vector stores), and it prepares the next one on the Model Context Protocol, which standardises the way tools are offered. Basic Python is enough. We will not call any model: the agent's "brain", in the experiments, is a hand-written function, and I remind you of it every time it matters. The examples use a neighbourhood library invented for the occasion, with invented data (the two book titles are kept in Italian: Il Gattopardo is The Leopard, Se questo e' un uomo is If This Is a Man).

ReAct: reasoning and acting in the same text

The name combines reasoning and acting. In the work that introduced it, Yao and colleagues (2022) have a language model generate, in an interleaved way, reasoning traces and task-specific actions: reasoning serves to build, follow and correct a plan and to handle exceptions, while actions make it possible to query external sources, such as a knowledge base or an environment, to gather new information. The resulting text has three kinds of line that repeat. Thought is the reasoning, Action is the tool to call with its argument, Observation is the result, and it is written by the program, never by the model. It closes with a Final Answer line.

The authors report in the abstract that on two interactive decision-making benchmarks (ALFWorld and WebShop) ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10%, with one or two in-context examples. Those are their numbers, on their benchmarks: they say nothing about how your task will go, and I report them as a fact of the literature, not as a promise.

To touch the mechanism we build the loop around a library question: how many copies of "Il Gattopardo" are available? A model cannot know that, but it can know how to break the problem into two lookups and a subtraction (total copies minus copies on loan). The listing that follows contains everything: two data tables, three tools, the action format, the executor and the loop, with three "policies" that produce ready-written text in place of the model.

"""A small ReAct loop. The "policy" is a hand-written function standing in
for the model: it shows the loop, it does not measure a model."""
import ast
import operator
import re

CATALOG = {"Il Gattopardo": 5, "Se questo e' un uomo": 2}
LOANS = {"Il Gattopardo": 3, "Se questo e' un uomo": 2}

OPS = {ast.Add: operator.add, ast.Sub: operator.sub,
       ast.Mult: operator.mul, ast.Div: operator.truediv}


def calc(expression):
    """Evaluates only numbers and + - * /: no eval()."""
    def ev(n):
        if isinstance(n, ast.Constant) and isinstance(n.value, (int, float)):
            return n.value
        if isinstance(n, ast.BinOp) and type(n.op) in OPS:
            return OPS[type(n.op)](ev(n.left), ev(n.right))
        raise ValueError("expression not allowed")
    return ev(ast.parse(expression, mode="eval").body)


TOOLS = {
    "catalog": lambda a: CATALOG[a],
    "loans": lambda a: LOANS[a],
    "calc": calc,
}
ACTION = re.compile(r"^Action: (\w+)\[(.*)\]$", re.M)


def execute(action_text):
    """The model proposes, the program executes: errors become observations."""
    m = ACTION.search(action_text)
    if not m:
        return "ERROR: invalid format, use Action: name[argument]"
    name, arg = m.groups()
    if name not in TOOLS:
        return f"ERROR: tool '{name}' does not exist, available: {sorted(TOOLS)}"
    try:
        return str(TOOLS[name](arg))
    except (KeyError, ValueError, SyntaxError, ZeroDivisionError) as e:
        return f"ERROR: {type(e).__name__}"


def valid_observations(transcript):
    every = re.findall(r"^Observation: (.*)$", transcript, re.M)
    return [o for o in every if not o.startswith("ERROR")]


def good_policy(transcript):
    ok = valid_observations(transcript)
    if len(ok) == 0:
        return "Thought: I need the total copies.\nAction: catalog[Il Gattopardo]"
    if len(ok) == 1:
        return "Thought: now the copies on loan.\nAction: loans[Il Gattopardo]"
    if len(ok) == 2:
        return f"Thought: I subtract.\nAction: calc[{ok[0]} - {ok[1]}]"
    return f"Thought: I have the number.\nFinal Answer: {ok[2]}"


def hallucinating_policy(transcript):
    if "Observation:" not in transcript:
        return "Thought: I search online.\nAction: web_search[Il Gattopardo copies]"
    return good_policy(transcript)


def stuck_policy(transcript):
    return "Thought: I need the total copies.\nAction: catalog[Il Gattopardo]"


def agent(policy, question, max_steps=6):
    transcript = f"Question: {question}"
    seen = []
    for step in range(1, max_steps + 1):
        output = policy(transcript)
        transcript += "\n" + output
        if "Final Answer:" in output:
            return step, output.split("Final Answer: ")[1], transcript
        action = ACTION.search(output)
        if action and action.group(0) in seen:
            return step, "INTERRUPTED: action repeated", transcript
        seen.append(action.group(0) if action else "")
        transcript += "\nObservation: " + execute(output)
    return max_steps, "INTERRUPTED: step budget exhausted", transcript


QUESTION = "How many copies of Il Gattopardo are available?"
for name, p in [("good", good_policy), ("hallucinating", hallucinating_policy),
                ("stuck", stuck_policy)]:
    steps, outcome, t = agent(p, QUESTION)
    print(f"--- {name} policy: {steps} steps, outcome: {outcome}")
    if name != "good":
        print(t)
print("--- hostile or wrong arguments")
for a in ['calc[__import__("os").getcwd()]', "calc[1/0]", "catalog[Missing Book]"]:
    print(a, "->", execute("Action: " + a))
expected = CATALOG["Il Gattopardo"] - LOANS["Il Gattopardo"]
print("independent check from the data:", expected)

# Output:
# --- good policy: 4 steps, outcome: 2
# --- hallucinating policy: 5 steps, outcome: 2
# Question: How many copies of Il Gattopardo are available?
# Thought: I search online.
# Action: web_search[Il Gattopardo copies]
# Observation: ERROR: tool 'web_search' does not exist, available: ['calc', 'catalog', 'loans']
# Thought: I need the total copies.
# Action: catalog[Il Gattopardo]
# Observation: 5
# Thought: now the copies on loan.
# Action: loans[Il Gattopardo]
# Observation: 3
# Thought: I subtract.
# Action: calc[5 - 3]
# Observation: 2
# Thought: I have the number.
# Final Answer: 2
# --- stuck policy: 2 steps, outcome: INTERRUPTED: action repeated
# Question: How many copies of Il Gattopardo are available?
# Thought: I need the total copies.
# Action: catalog[Il Gattopardo]
# Observation: 5
# Thought: I need the total copies.
# Action: catalog[Il Gattopardo]
# --- hostile or wrong arguments
# calc[__import__("os").getcwd()] -> ERROR: ValueError
# calc[1/0] -> ERROR: ZeroDivisionError
# catalog[Missing Book] -> ERROR: KeyError
# independent check from the data: 2

Four lines matter. The ACTION regex defines the format contract: if the model does not write Action: name[argument], the execute function does not guess, it returns an error observation. Still in execute, the name not in TOOLS check is what separates a real tool from one invented by the model. The calc function does not use eval but walks the syntax tree and admits only numbers and the four operations, so a hostile argument such as __import__("os") stops on a ValueError without executing anything. Finally agent has two brakes, max_steps and the seen list: remove them and the stuck policy consumes all the steps it is allowed; remove both and it never terminates.

Read the output as three stories. With the good policy four steps are needed (two lookups, a subtraction, the answer) and the result is 2, which matches the independent check done directly on the data at the end of the listing. The policy that invents the web_search tool receives an observation that lists the real tools and recovers, but pays one more step: five instead of four. The stuck policy repeats the same action and is stopped at the second attempt: without the brake you would not have a wrong answer, you would have a program that never terminates and a bill to pay at every turn.

Where an agent loop breaks

An agent's errors are not rare exceptions to handle in a corner: they are the normal path, and a good executor turns them into observations the model can read. Four cases turn up straight away, and the listing shows three of them.

• Non-existent tool. The model names something you did not declare; the executor answers with the list of valid tools.
• Wrong or hostile argument. A title that does not exist, a division by zero, an expression that tries to import a module: each produces an ERROR: ... line, not an exception that brings the program down.
• Repetition. The model rewrites the same thought and the same action without leaving the circle.
• Exhausted budget. The task does not converge within the number of steps allowed.

The four items have no order of priority: they are four different ways of going wrong, and each calls for a different countermeasure. The first two are fixed with a message that helps the model try again; the last two are not fixed, they are limited, because a model that repeats itself does not improve by insisting. In the ReAct paper the error analysis points to a typical pattern of the method, in which the model repeatedly generates the same thoughts and the same actions without leaving the circle, and the steps allowed are 7 on HotpotQA and 5 on FEVER, chosen because, the authors write, more steps did not improve performance. You find the right number for your task by measuring it, not by copying it. A caveat about my brake: seen forbids any repetition, and for a tool whose result changes over time (a counter, a state) it would be too strict. In that case count the consecutive repetitions and stop beyond a threshold.

Structured calls and training the model to use tools

Many model services offer a structured form of the loop: you declare the tools with a name, a description and a schema of the arguments, and the model answers with a JSON call instead of a line of text to be read with a regex. I do not describe any particular interface here, because the fields change from one provider to another and I have not verified them today. The principle holds for all of them: the structured call moves the format problem, not the truth problem. A well-formed JSON call can still name the wrong tool or pass an absurd argument, so the validation in execute stays where it is.

The Toolformer work (Schick and colleagues, 2023) tackles the same problem from another side: instead of writing the loop around a fixed model, it trains the model to decide by itself which interfaces to call, when, with which arguments and how to use the result, in a self-supervised way and with a handful of demonstrations for each tool. The tools in the experiment are a calculator, a question-answering system, two search engines, a translator and a calendar. For you the practical consequence is that the ability to choose the right tool can live in the model or in the prompt and the code around it, and your loop has to work in both cases.

Plan or react: an experiment on a grid

A ReAct agent decides one step at a time by looking at what is in front of it. When the task is linear that is enough and it costs little; when there are dead ends, something has to remember or predict. To see it without any model I build a tiny world: an 8 by 8 grid with obstacles (about 28% of the cells, with a fixed seed), from the top-left corner to the bottom-right one. Three agents tackle it on 200 different grids, all solvable. The first is purely reactive: it sees the free neighbours and goes to the one closest to the goal. The second has the same local view but remembers the cells it has seen and backtracks when it gets stuck. The third knows the whole map and plans with a breadth-first search.

"""React or plan: three agents on 200 8x8 grids with obstacles."""
import heapq
import random
from collections import deque

N = 8
MOVES = [(0, 1), (1, 0), (0, -1), (-1, 0)]  # fixed order: determinism


def make_grid(rng):
    while True:
        walls = {(r, c) for r in range(N) for c in range(N)
                if rng.random() < 0.28} - {(0, 0), (N - 1, N - 1)}
        if bfs(walls) is not None:
            return walls


def neighbors(p, walls):
    for dr, dc in MOVES:
        q = (p[0] + dr, p[1] + dc)
        if 0 <= q[0] < N and 0 <= q[1] < N and q not in walls:
            yield q


def bfs(walls):
    """Planner with the whole map: minimum length, or None."""
    queue, dist = deque([(0, 0)]), {(0, 0): 0}
    while queue:
        p = queue.popleft()
        if p == (N - 1, N - 1):
            return dist[p]
        for q in neighbors(p, walls):
            if q not in dist:
                dist[q] = dist[p] + 1
                queue.append(q)
    return None


def dijkstra(walls):
    """Second, independent calculation: the same result bfs is expected to give."""
    h, best = [(0, (0, 0))], {(0, 0): 0}
    while h:
        d, p = heapq.heappop(h)
        if p == (N - 1, N - 1):
            return d
        if d > best[p]:
            continue
        for q in neighbors(p, walls):
            if d + 1 < best.get(q, 10**9):
                best[q] = d + 1
                heapq.heappush(h, (d + 1, q))
    return None


def manhattan(p):
    return (N - 1 - p[0]) + (N - 1 - p[1])


def reactive(walls, limit=200):
    """Sees only the free neighbours, no memory: picks the closest to the goal."""
    p = (0, 0)
    for step in range(limit):
        if p == (N - 1, N - 1):
            return step
        p = min(neighbors(p, walls), key=manhattan, default=p)
    return None


def with_memory(walls, limit=200):
    """Same local view, but remembers the cells it has seen and backtracks (DFS)."""
    p, stack, seen, step = (0, 0), [(0, 0)], {(0, 0)}, 0
    while step < limit:
        if p == (N - 1, N - 1):
            return step
        fresh = sorted((q for q in neighbors(p, walls) if q not in seen),
                       key=manhattan)
        if fresh:
            p = fresh[0]
            seen.add(p)
            stack.append(p)
        else:
            stack.pop()
            if not stack:
                return None
            p = stack[-1]
        step += 1
    return None


rng = random.Random(7)
sample = [make_grid(rng) for _ in range(200)]
optimal = [bfs(m) for m in sample]
assert optimal == [dijkstra(m) for m in sample]

for name, f in [("reactive", reactive), ("with memory", with_memory)]:
    outcomes = [f(m) for m in sample]
    succeeded = [(e, o) for e, o in zip(outcomes, optimal) if e is not None]
    assert all(e >= o for e, o in succeeded)  # nobody beats the minimum
    extra = sum(e - o for e, o in succeeded) / len(succeeded)
    print(f"{name:12s} arrives in {len(succeeded):3d}/200, "
          f"extra steps over the minimum (mean over the successes): {extra:.1f}")
print(f"planner: 200/200, mean length {sum(optimal) / 200:.1f} "
      f"(minimum 14 for N=8)")

# Output:
# reactive     arrives in  37/200, extra steps over the minimum (mean over the successes): 0.0
# with memory  arrives in 200/200, extra steps over the minimum (mean over the successes): 6.9
# planner: 200/200, mean length 14.1 (minimum 14 for N=8)

In the listing, reactive chooses with min(..., key=manhattan) and has no state: inside a dead end it stays still or oscillates until the limit of 200 steps expires. with_memory keeps seen and stack, which are its "working memory" and its ability to go back. bfs starts from knowledge the others do not have, the complete map. As a second, independent calculation I rewrote the shortest-path search with Dijkstra's algorithm and the two lengths coincide on all 200 grids (the assert under the sample); a second check verifies that no agent beats the minimum.

The result reads like this. The agent without memory reaches the destination in only 37 grids out of 200, and in those 37 it does not waste a single step, because there was a path that always went downhill towards the goal. The agent with memory always arrives, but on average 6.9 steps above the minimum, which is the effort of discovering the way by walking it. The planner always arrives and in the fewest possible steps, provided it has the map: in real tasks that map is a model of the world (the schema of a database, the structure of a site) and it is never free. The experiment does not say how good a language model is at planning. It says that the shape of the loop, that is what the agent sees and what it remembers, decides a lot of the result before we even talk about models.

Minimal safety for an agent

An agent turns generated text into actions, and what is in the text is untrustworthy by definition. Five defences cost little and go in the loop, not in the prompt.

• A closed list of tools, declared by the program and not by the model.
• Validation of every argument before execution, with errors the model can read.
• A ceiling on the steps and a brake on repetitions.
• A human confirmation before every irreversible action or one with effects on the outside.
• Observations treated as data, never as instructions.

The last item is the least obvious: if a tool reads a page or a document that contains the sentence "ignore the previous instructions", that text enters the transcript like any other observation, and a model might treat it as an order. I do not demonstrate it with an experiment, because my policy is a function with no instructions to follow, and a test like that would say nothing about a real model. The countermeasure is structural: give the tools only the permissions the task needs, so that a hostile observation has nothing serious to make happen. In the article on the Model Context Protocol you will see how the standard treats user consent.

What this article does not prove

The "policies" in the listings are functions I wrote: they prove that the loop, the format and the brakes behave as described, they prove nothing about the behaviour of a real model or about its tendency to repeat itself or to invent tools. The figures on the benchmarks and on the steps allowed come from the ReAct paper and concern its tasks. The grid experiment compares three forms of agent on synthetic data, with a fixed seed, with the measurements reported above; with a bigger grid or a denser or sparser one the figures change, and the qualitative difference between reacting, remembering and planning is the only thing the experiment supports. To see how recent models spend more compute reasoning before answering, go to the article on reasoning models.

What you can do now

You can write the loop of an agent with a verifiable action format, an executor that turns errors into observations and two brakes against idle turns. You can read a benchmark result as a claim by the authors about a specific task, and you can measure with a small experiment the difference between reacting, remembering and planning. The natural next step is to ask how to offer tools to agents without rewriting the integration each time: the next article in the series covers it.