The answer is right, the path is not
The most annoying case in multi-agent systems is not the crash: it is the correct answer obtained the wrong way. A coordinator hands the work to a researcher, who queries a tool; the tool times out, the researcher retries, the second attempt succeeds and the writer delivers an impeccable text. The user sees nothing. You find out only, at the end of the month, that latency has doubled or that the costs do not add up. That is why observability in an agentic system is not for analyzing final errors: it is for reading the path.
The short answer is this: record every agent invocation and every tool call as a span of a trace, with the parent-child relationship, the duration and the outcome, and read the tree. OpenTelemetry is the neutral format for doing it, and it has conventions dedicated to agents, with a caveat that conditions everything else: they are in Development status. Below you will find a program that really produces a tree of spans and analyzes it; it calls no model.
What the conventions say for agents
The semantic conventions for generative AI describe names and attributes of spans that analysis tools know how to interpret. A useful detail: that work was moved from the main conventions repository to a dedicated repository, open-telemetry/semantic-conventions-genai, and the page on the official site now points there. The document on agent spans declares at the top Status: Development, so names and attributes may still change. In the documents I read on 13 November 2026 the following appears:
- the invocation of an agent or a workflow in your process is an
INTERNALspan withgen_ai.operation.nameequal toinvoke_agent(when a model drives the flow) or toinvoke_workflow(when the application defines the flow), and name{operation} {gen_ai.agent.name}; - the call of a tool is an
INTERNALspan with operationexecute_tool, nameexecute_tool {gen_ai.tool.name}and a mandatorygen_ai.tool.nameattribute; gen_ai.agent.namemust have low cardinality: an instance identifier does not belong in the name;- message content and system instructions are Opt-In attributes: they must not be recorded unless you decide so.
Interpret the list as a minimum contract. The first two points say how to name the spans, so that a tool recognizes them without configuration. The third keeps you from blowing up storage costs: a different agent name for every user produces millions of series. The fourth is a privacy choice stated plainly: prompts contain personal data and secrets. The document lists three ways of use: do not record instructions, inputs and outputs (the default), record them on the spans (suitable, the document says, for pre-production environments or where regulation does not apply), or store them elsewhere and record on the span only a reference, which is the way recommended in production when the volume is high or the data is sensitive.
A tree of spans you can run
The program simulates the example pipeline with three agents and a tool that fails once. It uses no models: I declare the times myself with a fake clock, which advances only when I say so, and this makes the output identical on every run; in a real system you will use the real clock and the times will vary. The trace_span context opens a span with an explicit start time, makes it current for the children with trace.use_span(..., end_on_exit=False), and in case of an exception records the exception, sets the status to error and adds error.type, then re-raises.
from collections import Counter, defaultdict
from contextlib import contextmanager
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import SpanKind, Status, StatusCode
exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("demo.agents")
_t = {"ms": 0} # fake clock: the times are the ones we declare
def advance(ms: int) -> None:
_t["ms"] += ms
def now_ns() -> int:
return _t["ms"] * 1_000_000
@contextmanager
def trace_span(name, kind=SpanKind.INTERNAL, **attributes):
span = tracer.start_span(name, kind=kind, attributes=attributes, start_time=now_ns())
with trace.use_span(span, end_on_exit=False):
try:
yield span
except Exception as exc: # record the exception and mark the span, then re-raise
span.record_exception(exc)
span.set_status(Status(StatusCode.ERROR, str(exc)))
span.set_attribute("error.type", type(exc).__name__)
raise
finally:
span.end(end_time=now_ns())
def run_tool(name, duration_ms, fails=False):
with trace_span(f"execute_tool {name}", **{"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": name, "gen_ai.tool.type": "function"}):
advance(duration_ms)
if fails:
raise TimeoutError(f"{name}: no reply after {duration_ms} ms")
def agent(name, work):
with trace_span(f"invoke_agent {name}", **{"gen_ai.operation.name": "invoke_agent",
"gen_ai.agent.name": name}):
work()
def researcher():
for attempt in (1, 2):
try:
run_tool("search_db", 800 if attempt == 1 else 120, fails=(attempt == 1))
break
except TimeoutError:
advance(50) # wait before the second attempt
def writer():
advance(300)
def coordinator():
agent("researcher", researcher)
agent("writer", writer)
with trace_span("invoke_workflow pipeline-report", **{"gen_ai.operation.name": "invoke_workflow"}):
agent("coordinator", coordinator)
spans = exporter.get_finished_spans()
by_id = {s.context.span_id: s for s in spans}
def depth(s):
return 0 if s.parent is None else 1 + depth(by_id[s.parent.span_id])
print(f"{len(spans)} spans, a single trace_id: {len({s.context.trace_id for s in spans}) == 1}")
for s in sorted(spans, key=lambda s: (s.start_time, depth(s))):
ms = (s.end_time - s.start_time) // 1_000_000
print(f"{' ' * depth(s) + s.name:<44}{ms:>5} ms {s.status.status_code.name}")
print("--- spans in error")
for s in spans:
if s.status.status_code is StatusCode.ERROR:
print(s.name, "|", s.attributes["error.type"], "|", s.status.description)
print("--- calls per tool")
for name, n in sorted(Counter(s.attributes["gen_ai.tool.name"] for s in spans
if s.attributes.get("gen_ai.operation.name") == "execute_tool").items()):
print(f"{name}: {n}")
print("--- parent time not covered by children")
children = defaultdict(list)
for s in spans:
if s.parent is not None:
children[s.parent.span_id].append(s)
for s in sorted(spans, key=lambda s: (s.start_time, depth(s))):
if s.context.span_id in children:
duration = (s.end_time - s.start_time) // 1_000_000
covered = sum((f.end_time - f.start_time) // 1_000_000 for f in children[s.context.span_id])
if duration != covered:
print(f"{s.name}: {duration - covered} ms outside every child span")
# Output:
# 6 spans, a single trace_id: True
# invoke_workflow pipeline-report 1270 ms UNSET
# invoke_agent coordinator 1270 ms UNSET
# invoke_agent researcher 970 ms UNSET
# execute_tool search_db 800 ms ERROR
# execute_tool search_db 120 ms UNSET
# invoke_agent writer 300 ms UNSET
# --- spans in error
# execute_tool search_db | TimeoutError | search_db: no reply after 800 ms
# --- calls per tool
# search_db: 2
# --- parent time not covered by children
# invoke_agent researcher: 50 ms outside every child span
In the tree you immediately read what the final result was hiding. The search_db tool appears twice under the researcher: the first lasting 800 ms and ending in error, the second lasting 120 ms. The researcher lasts 970 ms, the coordinator 1,270. The per-tool count says that search_db was called twice for a single request, and the error has its type and message ready. A successful retry is no news if nobody looks for it: that is why the output also counts the calls per tool.
The last section of the output is the most instructive. The researcher lasts 970 ms, but its two child spans cover 920: the missing 50 ms is the wait before the second attempt, which no span measured. A gap between parent and children is unobserved time, and in agentic systems that is often where queues, rate-limit waits or silent retries sit. The practical rule: if the sum of the children is much less than the parent, a span is missing, and it must be added before starting to optimize.
Sending the spans to a collector
InMemorySpanExporter is for tests. In operation the spans go to a collector with the OTLP protocol. The listing below builds the provider with a Resource that names the service, an OTLP exporter over HTTP and a BatchSpanProcessor. I ran it up to the construction, without creating spans: it sends nothing. The connection to a real collector is a sketch to adapt, because it requires a service listening that I did not start here.
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
provider = TracerProvider(resource=Resource.create({"service.name": "pipeline-report"}))
exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
print("endpoint:", exporter._endpoint)
provider.shutdown() # no span created: no sending starts
# Output:
# endpoint: http://localhost:4318/v1/traces
service.name is the property that separates your services in the collector's interface. The endpoint, if you do not pass it, comes from the variable OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and failing that is http://localhost:4318/v1/traces, which I read in the installed package. The BatchSpanProcessor accumulates spans and sends them in groups: in version 1.45.2 it waits up to 5 seconds, exports up to 512 spans at a time and keeps a queue of 2,048; if the queue fills up, the processor drops spans and writes it to the log. Better to know it before an incident: a burst of spans can vanish exactly when you would need it.
The collector can be Jaeger: its README says it accepts traces in OTLP over gRPC on port 4317 and over HTTP on 4318. If you find an old tutorial with opentelemetry.exporter.jaeger.thrift, know that the package opentelemetry-exporter-jaeger-thrift has as its latest version 1.21.0, of 7 November 2023, while the SDK is at 1.45.2: it is a stalled exporter, and the current path is OTLP.
What your framework offers
You do not always have to write the spans by hand. Every framework in the series has its own answer, and the differences matter:
- Mastra documents tracing as a built-in signal: hierarchical spans for agents, workflows, tools and model calls, with outcome fields (
success,error,suspend,limit,blockedand others) and a stop cause such asmax_steps. - LangGraph points in its documentation to the LangSmith path, which in the documented form requires an account and a key: the traces go to a LangChain service (I did not verify other forms of installation).
- CrewAI has its own anonymous telemetry on a private TracerProvider that never becomes the global one: its spans do not reach your collector, and yours do not reach CrewAI. Tracing for your backend goes through external integrations, which the documentation cites (Langfuse, Braintrust, Phoenix).
The criterion that emerges is where the spans end up. With Mastra they go into the storage you configure, and the connection to third-party systems is your choice; with LangSmith and with CrewAI's external integrations they go to a vendor, and what ends up in the traces, prompts and answers included, depends on the configuration: verify it. If the data is sensitive, choosing the tracing tool is a choice about data processing, not just about convenience: read the vendor's terms before turning on the export.
What goes wrong
I ran into a trap while building this series' examples. CrewAI's documentation says that the variable OTEL_SDK_DISABLED=true also turns off its telemetry, and I had set it for the CrewAI trials. Rerunning this article's program with that variable still in the environment, the span tree was empty: zero spans, no error. A global switch turns off the SDK for the whole process. When traces disappear, check the environment before the code.
Other recurring failures, worth putting in a checklist:
- Lost context: child spans attach to the parent only if the context reaches that far; a thread does not inherit it by itself and it must be passed explicitly.
- Too many spans: a loop without a cap produces one span per round; with LangGraph's default limit, 10007 supersteps, it is an enormous trace.
- Sensitive data in attributes: what is not Opt-In by convention does not belong in your attributes out of habit.
- High-cardinality names: user or request identifiers in the agent name blow up the storage.
These are four different mistakes with the same effect: telemetry becomes a cost without giving answers. The remedy is the same in all cases and is applied before turning on the export: decide which questions you want to be able to ask (which tool failed, how many times, where the time went) and record only what is needed to answer. To build the vocabulary of spans in a real system, the article on span design goes into detail.
What you can do now
You can read a span tree of a multi-agent system, find a hidden retry and a gap of unmeasured time, and you can write the spans of agents and tools with the names of the conventions, knowing that they are still in Development status. You know where to send the traces and what the choice implies, and you have a list of the failures that empty them. The series closes here: the choice of framework and the cost analysis gave the criteria, the memory of the agent with Letta and the Mastra framework the building blocks; with traces you can measure whether they work.
Frequently asked questions
Why OpenTelemetry and not an agent-specific tool?
OpenTelemetry is a neutral format: the spans you produce can be read by any collector that speaks OTLP, Jaeger included. The conventions for agents are in Development status, however, so pin the versions and reread the document when you update the SDK or the instrumentation library.
How do I trace a retry or a failure that then resolves?
Give each attempt its own span and mark the failed one as an error with record_exception and error.type, as in the program: then count the spans by tool name within a trace. Two calls of the same tool under the same agent are the signature of a retry, and the wait time between the two must be covered by a span of its own.
Can I record prompts in spans?
The conventions classify them as Opt-In attributes: the default way of use is to record neither instructions nor inputs nor outputs. For production they recommend storing the content elsewhere and putting only a reference on the span, so you can control access to the content separately. If you record them anyway, treat them as personal data and check the vendor's terms, if the collector is not yours.
Sources
- OpenTelemetry GenAI semantic conventions: agent spans - Development status; invoke_agent, invoke_workflow, span name, low cardinality of gen_ai.agent.name, consulted on 2026-11-13
- OpenTelemetry GenAI semantic conventions: execute tool - execute_tool {gen_ai.tool.name} span and attributes, consulted on 2026-11-13
- OpenTelemetry: "Moved" page of the GenAI conventions - the move to the semantic-conventions-genai repository, consulted on 2026-11-13
- Jaeger, README - OTLP over gRPC 4317 and HTTP 4318, consulted on 2026-11-13
- Mastra docs: Tracing - spans for agents, workflows, tools and model; outcome and stopReason fields, consulted on 2026-11-13
- CrewAI docs: Telemetry - anonymous telemetry, private TracerProvider, variables that turn it off, consulted on 2026-11-13
- LangGraph docs: Observability - documented tracing through LangSmith, with an account and key, consulted on 2026-11-13
Versions read from the registries on 2026-11-13: opentelemetry-sdk 1.45.2, opentelemetry-exporter-otlp-proto-http 1.45.2, opentelemetry-exporter-jaeger-thrift 1.21.0 (latest release 2023-11-07). The BatchSpanProcessor values (5000 ms, 512, 2048) and the default endpoint were read in the installed sources. The span program was run twice with the same output; the OTLP listing was run only up to the construction, without creating spans or opening connections; no collector was started.

Commenti
Caricamento commenti...
Sign in to comment