The problem: blind debugging in multi-agent systems

A typical agentic system composed of:

• Central orchestrator — coordinates agents, manages global state
• Specialized agents — data retrieval, filtering, generation, review
• Integrated tools — REST APIs, databases, search engines, on-prem LLM (Ollama)
• Task queue — async queues for long-running jobs

When a user submits a query, the flow passes through 10-20 components, often in parallel. If the result is wrong, the natural questions are: "At which step did the error occur? Which agent made the wrong decision? Which tool returned corrupted data? Which LLM hallucinated?". Without centralized tracing, you answer only by analyzing logs scattered across different files, hoping you remember which request-id each agent used.

End-to-end observability architecture

A complete observability solution for agentic systems requires three components:

• Distributed tracing — follow the flow of a single request through all agents and tools (OpenTelemetry + Jaeger)
• Structured logging — emit JSON logs with trace context, agent metadata, input/output state
• Metrics — end-to-end latency, LLM tokens consumed, tool success/failure rates, anomalies

The critical piece is the trace context — a unique ID and a parent-child ID that I propagate in every component, so all correlated events are linked together in a single view.

Setup OpenTelemetry + Jaeger locally

For a self-hosted agentic platform on a Hetzner VPS, we use OpenTelemetry (SDK) exported to local Jaeger. Jaeger requires no cloud subscription and runs in a Docker container.

# docker-compose.yml - add to monorepo
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"  # Jaeger UI
      - "6831:6831/udp"  # Jaeger agent (UDP)
    environment:
      - COLLECTOR_OTLP_ENABLED=true

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"

Jaeger exposes Prometheus metrics on localhost:14269/metrics. Prometheus scrapes and Grafana visualizes latency/error/throughput charts.

Instrument FastAPI with OpenTelemetry

In your FastAPI microservice (ml-service in the monorepo), add automatic instrumentation:

from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource

# Setup tracer provider
resource = Resource(attributes={
    SERVICE_NAME: "agentic-ml-service",
    "environment": "production",
    "platform": "ollama-rag"
})

jaeger_exporter = JaegerExporter(
    agent_host_name="localhost",
    agent_port=6831,
)

trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(BatchSpanProcessor(jaeger_exporter))
trace.set_tracer_provider(trace_provider)

# Instrument FastAPI with automatic traces
app = FastAPI()
FastAPIInstrumentor.instrument_app(app)
HTTPXClientInstrumentor().instrument()  # Trace HTTP client calls (e.g., to Ollama)
SQLAlchemyInstrumentor().instrument()  # Trace DB queries

tracer = trace.get_tracer(__name__)

With this configuration, every FastAPI request automatically generates a trace span. We add custom spans for agentic logic.

Instrument agents and tools

Every agent that calls an external tool (Ollama LLM, search API, database) must create spans to trace latency and results:

class ResearchAgent:
    def __init__(self):
        self.tracer = trace.get_tracer(__name__)

    async def search_documents(self, query: str, top_k: int = 5) -> List[str]:
        with self.tracer.start_as_current_span("research_agent.search_documents") as span:
            span.set_attribute("query", query)
            span.set_attribute("top_k", top_k)

            # Document search
            with self.tracer.start_as_current_span("redis_search") as search_span:
                results = await self.vectordb.query(query, top_k)
                search_span.set_attribute("results_count", len(results))

            return results

    async def generate_summary(self, documents: List[str]) -> str:
        with self.tracer.start_as_current_span("research_agent.generate_summary") as span:
            # Call Ollama LLM
            with self.tracer.start_as_current_span("ollama_llm_call") as llm_span:
                prompt = f"Summarize: {' '.join(documents)}"
                try:
                    response = await self.ollama_client.generate(
                        model="llama3.1:8b",
                        prompt=prompt
                    )
                    llm_span.set_attribute("model", "llama3.1:8b")
                    llm_span.set_attribute("tokens_generated", response.get("n_tokens", 0))
                    return response["text"]
                except Exception as e:
                    llm_span.record_exception(e)
                    span.add_event("llama_error", {"reason": str(e)})
                    raise

Each span includes attributes (query, model, result count), and errors are automatically logged in the trace. In the Jaeger UI you'll see the complete chain: orchestrator → agent.search → redis_search → redis_latency.

Structured logging with trace context

Plain logs are tedious for complex systems. Use structured logging (JSON) that includes trace-id automatically:

import logging
import json
from opentelemetry import trace

class StructuredLogFormatter(logging.Formatter):
    def format(self, record):
        # Capture current trace-id
        span = trace.get_current_span()
        trace_id = span.get_span_context().trace_id

        log_data = {
            "timestamp": record.created,
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
            "trace_id": f"{trace_id:032x}",  # 128-bit hex
            "span_id": f"{span.get_span_context().span_id:016x}",
        }

        # Include extra fields
        if hasattr(record, 'agent_id'):
            log_data["agent_id"] = record.agent_id
        if hasattr(record, 'tool_name'):
            log_data["tool_name"] = record.tool_name

        return json.dumps(log_data, default=str)

# Setup logger
logger = logging.getLogger("agentic-system")
handler = logging.StreamHandler()
handler.setFormatter(StructuredLogFormatter())
logger.addHandler(handler)

# Usage in agent code
def call_external_api(endpoint: str):
    logger.info(
        "Calling external API",
        extra={
            "tool_name": "search_api",
            "endpoint": endpoint,
            "agent_id": "research-001"
        }
    )

Now every log contains the trace-id. Elasticsearch / Loki ingests these JSON logs and you can search all events of a single request using the trace-id.

Key metrics for agentic systems

Beyond tracing and logging, monitor metrics to detect system-level anomalies:

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from prometheus_client import start_http_server

# Prometheus metrics
prometheus_reader = PrometheusMetricReader()
meter_provider = MeterProvider(metric_readers=[prometheus_reader])
meter = meter_provider.get_meter(__name__)

# End-to-end request latency
request_duration = meter.create_histogram(
    "request_duration_seconds",
    unit="s",
    description="End-to-end request latency"
)

# LLM tokens consumed per request
llm_tokens_consumed = meter.create_counter(
    "llm_tokens_consumed_total",
    unit="1",
    description="Total tokens sent to LLM"
)

# Tool call success rate
tool_success_rate = meter.create_counter(
    "tool_calls_total",
    unit="1",
    description="Tool invocations by status"
)

# Usage in request handler
@app.post("/chat")
async def chat_endpoint(req: ChatRequest):
    start = time.time()
    try:
        result = await orchestrator.run(req.query)
        duration = time.time() - start

        request_duration.record(duration)
        llm_tokens_consumed.add(result.get("tokens_used", 0))
        tool_success_rate.add(1, {"status": "success"})

        return result
    except Exception as e:
        tool_success_rate.add(1, {"status": "error", "error_type": type(e).__name__})
        raise

Graph these metrics in Grafana. Sudden increases in latency or drops in success rate signal issues (Ollama down, slow tool, overload).

Practical debugging: tracing a failed request

A real scenario: the user submits a query, the system returns the wrong result. Debugging workflow:

1. Open Jaeger UI (localhost:16686) and filter by service "agentic-ml-service"
2. Search the request timestamp (e.g., last 5 minutes)
3. Click on the trace — you'll see a waterfall with all spans:
   • orchestrator.run_query (5.2s total)
   • → research_agent.search_documents (1.3s)
   • → redis_search (0.8s)
   • → research_agent.generate_summary (3.1s)
   • → ollama_llm_call (2.9s)
   • → filtering_agent.rank_results (0.8s)
4. Inspect the filter agent — output showed low scores? Check the span attributes (score_threshold, results_before_filter, results_after_filter)
5. If the bug is in Ollama, click on ollama_llm_call — you'll see the prompt sent, token count, temperature, model. Replay the same query in Ollama manually and compare output.

This workflow is 100x faster than searching through aggregated logs.

Error handling and anomalies

Jaeger supports error visualization. When an exception is logged in a span, the UI shows that node in red. Add an anomaly detection layer:

async def detect_anomalies():
    """Monitor Prometheus metrics and alert on anomalies"""
    while True:
        await asyncio.sleep(60)  # Check every minute

        # High average latency: threshold 8 seconds
        avg_latency = await prometheus_client.query(
            'avg(rate(request_duration_seconds_sum[5m]))'
        )
        if avg_latency > 8:
            logger.warning("High latency detected", extra={"latency_s": avg_latency})
            # Send alert (email, Slack, etc.)

        # Error rate > 5%
        error_rate = await prometheus_client.query(
            'sum(rate(tool_calls_total{status="error"}[5m])) / sum(rate(tool_calls_total[5m]))'
        )
        if error_rate > 0.05:
            logger.error("High error rate", extra={"error_rate": error_rate})
            # Trigger incident

Best practices for agentic platforms observability

• Include agent decisions in spans — e.g. "agent chosen: research_agent", "fallback_strategy: use_cache". Helps understand the logical flow.
• Trace partial input/output — don't log the full text of documents (noise), but hash + length.
• Propagate trace-id cross-service — if Ollama runs on a different machine, pass the traceparent header in HTTP requests to Ollama.
• Sampling in production — tracing EVERY request can be expensive. Use probabilistic sampling (e.g. 10% in production, 100% in staging).
• Retention policy — Jaeger by default keeps traces for 72h. Configure it based on storage/costs.
• Alert on tool timeout — if a call to Ollama takes >30s, log an alert immediately (sign of overload or crash).

Roadmap: advanced observability (2026-2027)

• Anomaly correlation — if latency rises when tokens/min rises, suggest scaling Ollama
• Trace replay for testing — record input/output from a "correct" trace, and replay exactly the same tool calls for regression testing
• Native LLM observability — Ollama/Llama 3.x in the future may expose token distribution to detect hallucination (e.g. if token confidence drops below threshold)
• Intelligent distributed sampling — sample 100% of errors, 10% of successes, 1% of very-fast requests

FAQ

Q: Does local Jaeger have storage costs? Can I use it in production?

A: Yes, Jaeger is open-source and self-hosted — zero cloud costs. In production on a Hetzner VPS (CCX23, 16 GB RAM) you can run Jaeger in a "small" configuration that retains traces for 72-168h. For longer retention (1 year), integrate Cassandra as a storage backend (not covered here, but documented in Jaeger docs).

Q: How do I trace a tool that isn't HTTP (e.g. direct database query)?

A: Use OpenTelemetry's built-in instrumentations: SQLAlchemyInstrumentor for SQLAlchemy queries, or wrap code with tracer.start_as_current_span("db_query").

Q: Can I trace Ollama token cost in real-time for customer billing?

A: Yes — in the ollama_llm_call span record the `tokens_generated` field (provided by Ollama in response) and `tokens_input` (prompt length). Sum these in a Prometheus counter metric, and export to a billing system via API.

Q: Is tracing overhead significant?

A: Negligible (<2% latency added) with batch span processor. Tracing happens in a background thread. OpenTelemetry is optimized for production.

Q: If Jaeger goes down, does the system block?

A: No — Jaeger export is async and fault-tolerant. If the collector unreachable, spans are buffered and retried when online. If buffer exhausts memory, older spans are dropped (configurable).