Distributed Tracing: Understanding Request Flows
Distributed tracing is the technique that allows you to follow the complete path of a request through a distributed system, from the entry point to the final response. In a microservices architecture where a single user request can traverse 10, 20, or more services, distributed tracing is the only way to obtain an end-to-end view of the flow and identify bottlenecks, errors, and hidden dependencies.
Each request generates a trace, composed of a series of spans connected in a tree structure. Each span represents a unit of work: an HTTP call, a database query, a message published to a queue. Together, the spans tell the complete story of what happened during request processing.
What You Will Learn in This Article
- The structure of Trace and Span in OpenTelemetry
- The W3C Trace Context protocol for propagation
- Parent-child relationships and span links
- SpanKind: CLIENT, SERVER, PRODUCER, CONSUMER, INTERNAL
- Span attributes, events, and status
- Visualization patterns: waterfall and service map
Anatomy of a Trace
A trace is a directed acyclic graph (DAG) of spans representing the path
of a request. Each trace has a unique 128-bit identifier (trace_id) that is
propagated through all involved services. The initial span is called the root span
and represents the entire operation from start to finish.
Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736
[Root Span] API Gateway - POST /api/orders (250ms)
|
+-- [Child Span] Order Service - validate-order (15ms)
|
+-- [Child Span] Order Service - check-inventory (45ms)
| |
| +-- [Child Span] Inventory Service - GET /stock (40ms)
| |
| +-- [Child Span] Database - SELECT stock (8ms)
|
+-- [Child Span] Order Service - process-payment (120ms)
| |
| +-- [Child Span] Payment Service - charge (115ms)
| |
| +-- [Child Span] External API - Stripe charge (100ms)
|
+-- [Child Span] Order Service - send-confirmation (30ms)
|
+-- [Child Span] Notification Service - send-email (25ms)
Span Structure
A span represents a single operation within a trace. It contains all the information needed to understand what happened, how long it took, and whether errors occurred. Each span has a well-defined structure:
from opentelemetry import trace
from opentelemetry.trace import StatusCode, SpanKind
tracer = trace.get_tracer("order-service", "1.0.0")
def process_payment(order_id, amount, currency):
# Create a span with all components
with tracer.start_as_current_span(
name="process-payment",
kind=SpanKind.CLIENT, # Span type
attributes={ # Initial attributes
"order.id": order_id,
"payment.amount": amount,
"payment.currency": currency,
"payment.provider": "stripe"
}
) as span:
try:
# Add an event (automatic timestamp)
span.add_event("payment.validation.started", {
"validation.rules": "amount,currency,card"
})
validate_payment(amount, currency)
span.add_event("payment.validation.completed")
# Call to payment provider
result = stripe_client.charge(amount, currency)
# Add attributes after execution
span.set_attribute("payment.transaction_id", result.tx_id)
span.set_attribute("payment.status", "success")
# Set status
span.set_status(StatusCode.OK)
return result
except PaymentDeclinedException as e:
# Record exception as event
span.record_exception(e)
span.set_status(StatusCode.ERROR, "Payment declined")
span.set_attribute("payment.decline_reason", str(e))
raise
except TimeoutError as e:
span.record_exception(e)
span.set_status(StatusCode.ERROR, "Payment gateway timeout")
span.add_event("payment.retry.scheduled", {
"retry.attempt": 1,
"retry.delay_ms": 1000
})
raise
Span Components
| Component | Description | Example |
|---|---|---|
| trace_id | Unique trace ID (128 bit) | 4bf92f3577b34da6a3ce929d0e0e4736 |
| span_id | Unique span ID (64 bit) | 00f067aa0ba902b7 |
| parent_span_id | Parent span ID (empty for root) | a1b2c3d4e5f6a7b8 |
| name | Descriptive operation name | process-payment |
| kind | Span type (CLIENT, SERVER, etc.) | SpanKind.CLIENT |
| start_time | Start timestamp | 2026-02-17T10:30:00.000Z |
| end_time | End timestamp | 2026-02-17T10:30:00.120Z |
| attributes | Descriptive key-value pairs | payment.amount=99.99 |
| events | Timestamped point-in-time logs | payment.validation.completed |
| status | OK, ERROR, or UNSET | StatusCode.OK |
W3C Trace Context: The Propagation Standard
W3C Trace Context is the standard that defines how trace context is propagated across service boundaries. It specifies two HTTP headers that must be included in every inter-service request:
# W3C Trace Context Headers
# traceparent: contains trace_id, parent_span_id, trace_flags
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
| | | |
v v v v
version trace_id (32 hex) parent_span_id (16 hex) flags (sampled=01)
# tracestate: vendor-specific state (optional)
tracestate: congo=t61rcWkgMzE,rojo=00f067aa0ba902b7
When a service receives a request with the traceparent header, it extracts the
trace_id and parent_span_id, creates a new child span with the same
trace_id, and propagates the updated context to subsequent calls. This mechanism
ensures that all spans of a request share the same trace_id.
SpanKind: Classifying Operations
SpanKind classifies the role of a span in the request flow. This information is fundamental for backends that need to reconstruct system topology and calculate inter-service latencies.
from opentelemetry.trace import SpanKind
# SERVER: service receives a request from a remote client
with tracer.start_as_current_span("handle-request", kind=SpanKind.SERVER):
# Handle incoming HTTP request
pass
# CLIENT: service sends a request to a remote service
with tracer.start_as_current_span("call-payment-api", kind=SpanKind.CLIENT):
# HTTP call to payment-service
pass
# PRODUCER: service sends a message to a broker
with tracer.start_as_current_span("publish-order-event", kind=SpanKind.PRODUCER):
# Publish to Kafka topic
pass
# CONSUMER: service receives a message from a broker
with tracer.start_as_current_span("process-order-event", kind=SpanKind.CONSUMER):
# Consume from Kafka topic
pass
# INTERNAL: internal service operation (default)
with tracer.start_as_current_span("validate-order", kind=SpanKind.INTERNAL):
# Internal validation logic
pass
Rules for Choosing SpanKind
- SERVER: use when your service is the recipient of a synchronous request (HTTP, gRPC)
- CLIENT: use when your service sends a synchronous request to another service
- PRODUCER: use when your service produces an asynchronous message (Kafka, RabbitMQ, SQS)
- CONSUMER: use when your service consumes an asynchronous message
- INTERNAL: use for internal operations that do not cross network boundaries
Span Links: Connecting Different Traces
Beyond parent-child relationships, OTel supports Span Links for connecting spans belonging to different traces. Links are useful in asynchronous scenarios where an operation is caused by another but is not a direct child. For example, a message in a queue can be linked to the span that produced it, even if the consumer's trace is different.
from opentelemetry import trace, context
# Scenario: batch processing messages from a queue
# Each message has its own original trace context
def process_batch(messages):
# Create links to all messages in the batch
links = []
for msg in messages:
# Extract trace context from message
msg_context = extract_context(msg.headers)
span_context = trace.get_current_span(msg_context).get_span_context()
links.append(trace.Link(
context=span_context,
attributes={"messaging.message.id": msg.id}
))
# Create a span with links to all original messages
with tracer.start_as_current_span(
name="process-message-batch",
kind=SpanKind.CONSUMER,
links=links,
attributes={
"messaging.batch.message_count": len(messages)
}
) as span:
for msg in messages:
process_single_message(msg)
Trace Visualization
Distributed traces are typically visualized in two complementary ways:
Waterfall View (Gantt Chart)
The waterfall view shows spans in temporal order, with indentation for parent-child relationships. It allows immediately identifying where time is spent and which operations are sequential vs parallel. It is the most used view for debugging individual slow requests.
Service Map (Topology)
The service map shows dependencies between services as a directed graph. Each node represents a service, each edge a communication. Aggregated metrics (latency, error rate, throughput) are overlaid on edges. It is the ideal view for understanding overall architecture and identifying critical services or failure points.
Best Practices for Distributed Tracing
Name spans with operations, not URLs: use create-order, not
POST /api/v1/orders. High-cardinality URLs (with IDs) create aggregation problems.
Add business-relevant attributes: order.id, user.tier,
payment.method transform traces from technical tools into business analysis tools.
Record exceptions with record_exception: automatically captures error type, message,
and stacktrace as a span event.
Use SpanKind correctly: allows backends to calculate network latencies
(difference between CLIENT span and corresponding SERVER span).
Conclusions and Next Steps
Distributed tracing is the fundamental tool for understanding distributed system behavior. The combination of trace context propagation (W3C), structured spans with semantic attributes, and waterfall/service map visualizations provides the visibility needed to diagnose complex problems that span service boundaries.
The correct choice of SpanKind, the use of meaningful attributes, and the recording of events and exceptions transform traces from simple timelines into powerful analysis tools that tell the complete story of every request.
In the next article, we will explore auto-instrumentation, the technique that allows obtaining distributed traces without modifying application code, using agents and automatic instrumentation libraries for Java, Python, and Node.js.