06 - Multi-Agent Orchestration: Production Architectures

The problem: three agents in the prototype, a dozen in production

A prototype with two or three agents almost always works, because the person watching it acts as the supervisor: they see the loop, interrupt, restart. In production that supervisor is gone, and the annoyances become failures: two agents bouncing the same request back and forth, a conversation that never ends, an error that reaches the final answer intact, an unexpected token bill. This article gives you a criterion for deciding whether you need a multi-agent system, describes the five ways of coordinating the work and the three architectures for connecting the agents, and then four defenses that hold whatever you choose: a cap on every loop, a log of what happened, retries with a circuit breaker, traces.

The listings are run locally with only the Python 3.14.4 standard library, and the agents are fake functions that follow a fixed rule: no model called, no broker or database started. They show the mechanics of orchestration, not the behavior of a real model. If you have not read how a single agent reasons, start from Fundamentals: OODA Loop, ReAct and Tool Calling; the frameworks compared earlier are LangChain and LangGraph, CrewAI and AutoGen.

Before orchestrating: do you need more than one agent?

Microsoft's guide to orchestration patterns opens with a ladder of complexity: a direct call to the model, a single agent with tools, a multi-agent orchestration. The advice is to use the lowest level that meets the requirements reliably, because every level adds coordination, latency and cost. The single agent with tools is indicated as often the right choice for enterprise cases, simpler to debug and to test. What justifies the jump is a prompt too complex for a single agent, too many tools at once, or different security boundaries for each agent.

Two sources measure the cost of the jump. Anthropic, describing its own research system with a coordinator agent and subagents in parallel, reports that in its data agents use about 4 times the tokens of a chat and multi-agent systems about 15 times, and concludes that you need tasks of high enough value to pay for it. The paper “Towards a Science of Scaling Agent Systems” (260 configurations, six benchmarks, five architectures) measures errors instead: in its tests independent agents, without central verification, amplify trace errors by 17.2 times compared to a single agent; a centralized coordination that validates before aggregating keeps them at 4.4 times; decentralized and hybrid sit in between (7.8 and 5.1). The authors also find that where the single agent already exceeds 45% accuracy, adding agents gives negative returns.

Those figures describe the authors' benchmarks, not your system, and must not be copied as thresholds. They do say, though, that the ways a multi-agent system goes wrong are predictable and come down to a short list.

• a loop with no cap (a handoff that bounces, a conversation that does not converge, an agent that calls the same tool again);
• an error that nobody checks and that passes intact from one agent to the next;
• a context that grows at every handoff, with cost and latency growing together;
• a failure of one component (model, tool, broker) that spreads to the others;
• a behavior you cannot reconstruct afterwards, because nobody recorded what happened.

The five points are independent and the order does not matter, but they do not weigh the same: the first two are design flaws prevented by rules in the code (caps, verification), the third by the choice of what to pass from one agent to the other, the last two by infrastructure (retries, breakers, traces). The rest of the article follows this order.

The five orchestration patterns

Microsoft's guide names five patterns, each with its synonyms in use in the field. The table lists them with the rule, taken from the same guide, for recognizing when to avoid them.

The middle column is the one that matters: in the sequential and the concurrent the order is decided by your code and the system stays predictable; in the other three it is decided by an agent, that is, a model, and predictability has to be recovered with caps and checks. The pattern that many tutorials call “plan-first” corresponds to the fifth: the guide calls it magentic and describes it with a manager that builds a ledger of tasks and progress. They are the classification of one guide, not a standard: other sources use other names for the same things.

Sequential and concurrent: you decide the order

The block builds two functions on top of fake agents, each one a rule that wraps the text it receives in a letter. sequential passes the output of each to the input of the next and keeps a trace; concurrent gives the same input to all with a ThreadPoolExecutor and collects the results with pool.map, which returns them in submission order and not in completion order. It is not cosmetic: aggregating in arrival order, the same request would give different syntheses from one run to the next.

from concurrent.futures import ThreadPoolExecutor

def agent(name, reply):
    """A fake agent: no model, it answers with a fixed rule."""
    def run(text):
        return f"{reply}({text})"
    run.name = name
    return run

# Sequential: the code decides the order, not the agents
def sequential(agents, text):
    trace = []
    for a in agents:
        text = a(text)
        trace.append((a.name, text))
    return text, trace

outcome, trace = sequential(
    [agent("draft", "B"), agent("review", "R"), agent("style", "S")], "topic")
for name, output in trace:
    print(f"{name:10} -> {output}")

# Concurrent: same input to everyone, results collected in submission order
def concurrent(agents, text):
    with ThreadPoolExecutor(max_workers=len(agents)) as pool:
        return dict(zip((a.name for a in agents), pool.map(lambda a: a(text), agents)))

print(concurrent([agent("fundamentals", "F"), agent("technical", "T"),
                  agent("sentiment", "M")], "ACME"))

# Output:
# draft      -> B(topic)
# review     -> R(B(topic))
# style      -> S(R(B(topic)))
# {'fundamentals': 'F(ACME)', 'technical': 'T(ACME)', 'sentiment': 'M(ACME)'}

In the first part of the output the last agent receives the work of all the previous ones: it is the advantage (every step sees the full result of the previous one) and the risk (if the first one is wrong, the others work on the error, as the guide notes among the reasons to avoid the pattern). In the second the agents do not see each other, and the question to ask is who resolves the conflict when two results contradict each other. With real agents the time of the concurrent one tends to be that of the slowest and not the sum, while the tokens add up; I did not measure it here, because I did not call models.

Handoff and group chat: you need caps

In the handoff an agent receives the request and decides who to pass it to: in customer support the first one sorts, the specialist answers. The trouble is the bounce, two agents sending the request back to each other because each believes it belongs to the other. The block shows the defect on purpose: legal sends back to accounting, which sends back to legal; the max_hops cap interrupts the loop and the result says what happened (stopped) instead of returning an invented answer.

def triage(request, hops):
    """Routes: returns (next_agent, None) or (None, answer)."""
    if "invoice" in request:
        return "accounting", None
    return None, "Generic answer from triage"

def accounting(request, hops):
    if "refund" in request:
        return "legal", None            # needs another specialist
    return None, "Invoice found"

def legal(request, hops):
    return "accounting", None           # deliberate defect: sends it back

AGENTS = {"triage": triage, "accounting": accounting, "legal": legal}

def handoff(request, max_hops=4):
    current, chain = "triage", []
    while len(chain) < max_hops:
        chain.append(current)
        nxt, answer = AGENTS[current](request, chain)
        if answer is not None:
            return {"chain": chain, "answer": answer}
        current = nxt
    return {"chain": chain, "answer": None, "stopped": "hop cap"}

print(handoff("where is my invoice?"))
print(handoff("invoice with refund"))

# Output:
# {'chain': ['triage', 'accounting'], 'answer': 'Invoice found'}
# {'chain': ['triage', 'accounting', 'legal', 'accounting'], 'answer': None, 'stopped': 'hop cap'}

The first request closes in two hops. The second reaches the cap with the chain triage, accounting, legal, accounting, and the None in answer matters: whoever calls knows there is no answer and can pass to a person instead of showing any old text. The cap of 4 is arbitrary: choose yours by looking at the longest legitimate chains you observe, plus a margin. The same logic holds for the group chat, where the guide suggests three agents or fewer and, in maker-checker loops (one proposes, one verifies), a maximum number of iterations with a fallback, for example escalation to a person. The equivalent parameters in AutoGen are in the article on AutoGen.

Plan and ledger: the magentic pattern

When you do not know the steps in advance, an agent can build a plan and a manager executes it while keeping a ledger: what is decided, done, blocked. The block isolates the deterministic part, the execution of a plan with dependencies. The “planner” is a constant (PLAN) because the part that requires a model cannot be run here; the execution uses graphlib.TopologicalSorter, and a task blocked by a failed predecessor is skipped and noted, not run in a vacuum.

from graphlib import TopologicalSorter

# The fake "planner" always returns the same plan: task -> dependencies
PLAN = {
    "collect_data": [],
    "analyze_a": ["collect_data"],
    "analyze_b": ["collect_data"],
    "summary": ["analyze_a", "analyze_b"],
}
FAILURES = {"analyze_b"}                 # the task that fails here

def run_plan(plan, failures):
    ledger = {name: "to do" for name in plan}
    ts = TopologicalSorter(plan)
    ts.prepare()
    while ts.is_active():
        for name in sorted(ts.get_ready()):
            blocked = [d for d in plan[name] if ledger[d] != "done"]
            if blocked:
                ledger[name] = f"skipped (depends on {', '.join(blocked)})"
            elif name in failures:
                ledger[name] = "failed"
            else:
                ledger[name] = "done"
            ts.done(name)
    return ledger

for name, status in run_plan(PLAN, FAILURES).items():
    print(f"{name:14} {status}")

# Output:
# collect_data   done
# analyze_a      done
# analyze_b      failed
# summary        skipped (depends on analyze_b)

The ledger says that summary was not run because analyze_b failed, and it does not produce a synthesis built on half of the data. If you choose to deliver a partial result anyway, you must declare it to the user: it is a product decision. The cost of the pattern is its reason for being: building and discussing a plan takes time, and the guide advises against it with tight deadlines or when the path is already known.

Where the control sits: hub, peers or queue

The patterns say how the work is coordinated; the architectures say how the agents are connected. There are three, and they are not an official classification: they are the forms met most often, and they can be combined.

In hub-and-spoke a coordinator agent talks to all the others, which do not talk to each other: it is the form of the orchestrator with subagents described by Anthropic for its research system. Everything goes through one point, so logging, limiting and stopping are easy; for the same reason the coordinator is a point of failure and a bottleneck. Anthropic notes that its lead agents wait for the subagents synchronously, which simplifies coordination but creates bottlenecks, and that going asynchronous would bring problems of state consistency and error propagation.

In peer-to-peer the agents talk to each other directly. I have no listing for this form: it would require discovery and routing between peers, which makes no sense to simulate in thirty lines and which Microsoft's guide names only as a “peer-based protocol”. There is a source, though: in the scaling paper the decentralized architecture (agents exchange messages as peers) amplifies errors more than the centralized one, 7.8 against 4.4 times in their experiments, because a control point is missing. It is not a ranking: on some benchmarks the decentralized one turned out to be the best, and the authors conclude that what counts is the match between the shape of the communication and the structure of the task.

In the message queue the agents communicate by publishing to and reading from a broker: the sender does not know who will read, and the reader may be off when the message arrives. The listing is an in-memory broker with a few methods, useful for seeing three properties and not for replacing RabbitMQ or Kafka (which I did not start): a message stays in the queue until someone reads it, one that fails comes back with a counter, and after two attempts it ends up in a “dead letter queue” instead of blocking the others.

from collections import defaultdict, deque

class Broker:
    """A minimal in-memory broker: queues per topic, attempts, dead letter queue."""
    def __init__(self, max_attempts=2):
        self.queues = defaultdict(deque)
        self.dead = []
        self.max_attempts = max_attempts

    def publish(self, topic, message):
        self.queues[topic].append({"body": message, "attempts": 0})

    def consume(self, topic, handler):
        """Processes everything in the queue; a message that fails goes back in."""
        queue = self.queues[topic]
        for _ in range(len(queue)):
            msg = queue.popleft()
            try:
                handler(msg["body"])
            except ValueError as error:
                msg["attempts"] += 1
                if msg["attempts"] >= self.max_attempts:
                    self.dead.append((msg["body"], str(error)))
                else:
                    queue.append(msg)

def analyst(body):
    if body == "malformed":
        raise ValueError("invalid JSON")
    print("analysis of:", body)

b = Broker()
b.publish("analysis", "order 17")      # the analyst is "off": the message waits
b.publish("analysis", "malformed")
print("queued before consumption:", len(b.queues["analysis"]))
for round_no in (1, 2):
    b.consume("analysis", analyst)
    print(f"round {round_no}: queued {len(b.queues['analysis'])}, dead {len(b.dead)}")
print("dead letter queue:", b.dead)

# Output:
# queued before consumption: 2
# analysis of: order 17
# round 1: queued 1, dead 0
# round 2: queued 0, dead 1
# dead letter queue: [('malformed', 'invalid JSON')]

The message order 17 was in the queue before the analyst existed and is processed in the first round; the malformed one fails, goes back in, fails again and ends up in dead with the reason. Without the attempt limit it would spin forever: the handoff bounce in its infrastructure version. A real broker adds what is missing here, persistence on disk and acknowledgment of receipt; it does not solve ordering, though: two agents reading from the same queue do not see the messages at the same instant, and the shared state becomes your problem again.

The table attaches no numbers (how many agents, how many milliseconds) because I have neither a source nor a measurement that would hold up. The rule I draw from it: start from the hub, the form that is easiest to stop and observe, and move to the queue when you have a precise reason, for example components to switch off or scale separately. Peer-to-peer makes sense only if the absence of a coordinator is a requirement, not a convenience.

State: an event log and sagas

With several agents writing, shared state is where bugs hide. One approach is not to save the current state but the events that produced it: it is Event Sourcing, with events recorded in an append-only store and the state derived by replaying them. A second pattern is needed when an operation touches several agents and must succeed entirely or not at all: the saga splits the operation into local transactions, each with its own compensation to run if a later step fails. Both are in the pattern catalog of the Azure Architecture Center.

The block combines them in a travel agent: booking flight, hotel and car, with the car unavailable. saga runs the steps in order and, at the first failure, the compensations in reverse order, only for the steps that already succeeded. Every action writes an event on the log; state_at(n) rebuilds the active bookings by replaying the first n events.

ledger = []                         # event log: append only, never edited

def event(kind, **data):
    ledger.append({"n": len(ledger) + 1, "type": kind, **data})

def book(resource, fails=False):
    if fails:
        raise RuntimeError(f"{resource} unavailable")
    event("booked", resource=resource)

def cancel(resource):
    event("cancelled", resource=resource)

# Each step is (action, compensation): the compensation is not a magic "undo",
# it is an action written on purpose to undo that step.
STEPS = [("flight", False), ("hotel", False), ("car", True)]

def saga(steps):
    done = []
    for resource, fails in steps:
        try:
            book(resource, fails)
            done.append(resource)
        except RuntimeError as error:
            event("failed", reason=str(error))
            for already in reversed(done):      # compensations in reverse order
                cancel(already)
            return False
    return True

print("saga completed:", saga(STEPS))
for e in ledger:
    print(e)

# The current state is not stored: it is rebuilt by replaying the log
def state_at(n):
    active = set()
    for e in ledger[:n]:
        if e["type"] == "booked":
            active.add(e["resource"])
        elif e["type"] == "cancelled":
            active.discard(e["resource"])
    return sorted(active)

print("after event 2:", state_at(2))
print("after event 5:", state_at(5))

# Output:
# saga completed: False
# {'n': 1, 'type': 'booked', 'resource': 'flight'}
# {'n': 2, 'type': 'booked', 'resource': 'hotel'}
# {'n': 3, 'type': 'failed', 'reason': 'car unavailable'}
# {'n': 4, 'type': 'cancelled', 'resource': 'hotel'}
# {'n': 5, 'type': 'cancelled', 'resource': 'flight'}
# after event 2: ['flight', 'hotel']
# after event 5: []

The log tells everything: two bookings, a failure, two cancellations in reverse order. Replayed up to event 2 it gives hotel and flight, up to 5 nothing: looking at the state at a past instant is what you buy with Event Sourcing, and you pay for it by replaying the events (which is why the catalog mentions materialized views). The example hides one point: real compensations can fail, and the catalog warns that the steps after the point of no return (the “pivot”) must be safely repeatable, that is, idempotent. Moreover, not everything can be undone: a message already sent to a customer is compensated with another message.

When something breaks: retries, breaker, fallback

The expected failures of an agent are those of any remote call: the model does not answer, the tool times out, the output does not have the expected format. Microsoft's catalog covers two defenses that combine. The retry tries again waiting longer at each attempt, on the assumption that the error is transient; the guide warns that an aggressive policy can make an already struggling service worse, and that retrying is safe only if the operation is idempotent. The circuit breaker does the opposite: after too many errors it stops calling for a period, then lets a trial through. It has three states: closed (calls go through), open (they fail immediately), half-open (one trial: if it succeeds it closes again, if it fails it reopens).

The block implements both on a fake clock that advances only when you tell it to, so the output does not depend on the machine. CircuitBreaker.call is the state machine; retry waits a time chosen at random between 0 and the minimum of the cap and base * 2 ** n (full jitter, which prevents many clients from retrying at the same instant), with a fixed seed. One line counts more than the others, except CircuitOpen: raise: retrying against an open circuit defeats the breaker, and Microsoft's page says the retry must stop when the breaker signals a non-transient fault.

import random

class Clock:
    """Fake time: it advances only when you say so, so the output is always the same."""
    def __init__(self):
        self.t = 0.0
    def sleep(self, seconds):
        self.t += seconds

class CircuitOpen(Exception):
    pass

class CircuitBreaker:
    def __init__(self, clock, threshold=3, pause=30.0):
        self.clock, self.threshold, self.pause = clock, threshold, pause
        self.state, self.errors, self.opened_at = "closed", 0, None

    def call(self, function):
        if self.state == "open":
            if self.clock.t - self.opened_at < self.pause:
                raise CircuitOpen("circuit open: call not attempted")
            self.state = "half-open"        # lets ONE trial through
        try:
            result = function()
        except ConnectionError:
            self.errors += 1
            if self.state == "half-open" or self.errors >= self.threshold:
                self.state, self.opened_at = "open", self.clock.t
            raise
        self.state, self.errors = "closed", 0
        return result

def retry(function, clock, waits, attempts=4, base=1.0, cap=8.0, seed=7):
    rng = random.Random(seed)
    for n in range(attempts):
        try:
            return function()
        except CircuitOpen:
            raise                              # no retrying against an open circuit
        except ConnectionError:
            if n == attempts - 1:
                raise
            wait = rng.uniform(0, min(cap, base * 2 ** n))   # "full" jitter
            waits.append(round(wait, 2))
            clock.sleep(wait)

def agent_down():
    raise ConnectionError("model unreachable")

clock = Clock()
breaker = CircuitBreaker(clock)
waits = []
try:
    retry(lambda: breaker.call(agent_down), clock, waits)
except (ConnectionError, CircuitOpen) as e:
    print("retry finished:", type(e).__name__, "| waits:", waits, "| state:", breaker.state)
try:
    breaker.call(agent_down)
except CircuitOpen as e:
    print("right after:", e)
clock.sleep(31)
print("after 31 s, a successful trial:", breaker.call(lambda: "ok"), "| state:", breaker.state)

# Output:
# retry finished: CircuitOpen | waits: [0.32, 0.3, 2.6] | state: open
# right after: circuit open: call not attempted
# after 31 s, a successful trial: ok | state: closed

The retry waited three times (0.32, 0.3 and 2.6 seconds in fake time); at the third error the circuit opened and the fourth attempt did not start. Right after, a call is rejected without trying; after 31 seconds, beyond the pause of 30, a trial goes through, succeeds and the circuit closes again. The thresholds (3 errors, 30 seconds, 4 attempts, a cap of 8 seconds) are examples: they depend on the recovery time of the service and on the cost of an error, and have to be measured on yours. With a paid model, a cap on the total number of calls per request counts as much as the rest.

The last defense is the fallback (graceful degradation): when a component does not answer, the system returns something less good but declared, for example a simpler or rule-based agent in place of the primary one, a partial result with a warning, a cached answer labeled as such. Anthropic describes the variant for long-running systems: checkpoints to resume from where the agent had got to, because restarts are costly, and telling the agent that a tool is not working and letting it adapt, which according to them works surprisingly well. The line not to cross: do not hide from the user that the answer is degraded.

Observing: traces, cost, your own thresholds

A non-deterministic multi-agent system can be debugged only if you record what it did. Anthropic writes that users reported agents “not finding obvious information” and that only the full trace in production made it possible to tell whether it was a wrong query, a badly chosen source or a broken tool. The idea is that of traditional systems: every request has an identifier, every step is a span with a parent, and the tree shows where time and errors accumulate.

The block builds a minimal tracer with a context manager: each span notes parent, declared duration and attributes, and if an exception rises inside it, it records the error type and lets it rise. The example has a planner and a search agent whose tool times out; the tokens are numbers invented for the example, not measurements.

from contextlib import contextmanager

class Tracer:
    """Keeps the list of spans. Time is fake: every event costs what you declare."""
    def __init__(self):
        self.spans, self.stack, self.now = [], [], 0.0

    @contextmanager
    def span(self, name, duration, **attributes):
        parent = self.stack[-1] if self.stack else None
        s = {"id": len(self.spans) + 1, "parent": parent, "name": name,
             "start": self.now, "attributes": attributes}
        self.spans.append(s)
        self.stack.append(s["id"])
        try:
            yield s
        except Exception as e:
            s["attributes"]["error"] = type(e).__name__
            raise
        finally:
            self.now += duration
            s["end"] = self.now
            self.stack.pop()

tr = Tracer()
try:
    with tr.span("request", 0.1, trace="T-1"):
        with tr.span("agent:planner", 1.5, tokens=420):
            pass
        with tr.span("agent:search", 4.0, tokens=1800):
            with tr.span("tool:search", 2.0):
                raise TimeoutError("the tool does not respond")
except TimeoutError:
    pass

parents = {s["id"]: s["parent"] for s in tr.spans}
def depth(i):
    return 0 if parents[i] is None else 1 + depth(parents[i])
for s in tr.spans:
    duration = round(s["end"] - s["start"], 1)
    print("  " * depth(s["id"]) + f'{s["name"]} {duration}s {s["attributes"]}')
print("total tokens:", sum(s["attributes"].get("tokens", 0) for s in tr.spans))

# Output:
# request 7.6s {'trace': 'T-1', 'error': 'TimeoutError'}
#   agent:planner 1.5s {'tokens': 420}
#   agent:search 6.0s {'tokens': 1800, 'error': 'TimeoutError'}
#     tool:search 2.0s {'error': 'TimeoutError'}
# total tokens: 2220

The tree shows at once where the error was born (tool:search) and how it climbs: whoever looks only at the outer level sees “request failed” without knowing why. The duration of a span includes that of its children, so to know where the time was spent it has to be subtracted. In a real system you do not write a tracer of your own: you use OpenTelemetry. Its conventions for generative AI have been moved to the semantic-conventions-genai repository; I did not compare them with the listing, so consult that repository for the attribute names before adopting them.

What to record in a span depends on the system, but some quantities are almost always needed:

• the request identifier, the same in all the spans of the same job;
• the agent or the tool, and the version of the prompt or model used;
• the input and output tokens, to reconstruct the cost of every request;
• the outcome (ok, error, skipped) and the type of error;
• the number of hops and of attempts, to see the bounces and the retries.

From these quantities you derive the metrics that matter: latency per agent, error rate, tokens per request, requests that reach the cap, breaker openings. Alarm thresholds are not copied from an article: they are set by observing a period of normal operation of your own system. An alarm on “requests that reached the hop cap” or on “breaker openings” needs little data and tells you right away that something is wrong.

Where to go from here

How to measure that the whole works, and not just that it runs, is in Testing and Evaluation; the memory the agents pass to each other is in Memory Systems; the risk of agents that trust one another in Security and Safety; how to design the tools they call in Advanced Tool Calling.

Frequently asked questions

What you can do now

You can decide whether a problem requires several agents or whether one with tools is enough, and choose one of the five patterns according to who decides the next step. You can put a cap on a handoff and on a group chat, run a plan with dependencies leaving a ledger of what failed or was skipped, connect the agents with a hub or with a queue knowing what each form costs you, rebuild a state from an event log and compensate a saga. You can combine retries and breaker without making them fight, record traces from which you can tell where an error was born, and set the alarm thresholds on your data and not on a table.