Backend Integration: Jaeger, Prometheus, and Grafana
After instrumenting applications and configuring the Collector, the next step is connecting telemetry data to storage and visualization backends. The most popular open source stack combines Jaeger for distributed traces, Prometheus for metrics, and Grafana as the unified visualization layer. Together, these tools form a complete, production-ready observability system.
In this article, we will configure the entire stack from scratch, from Docker Compose deployment to creating Grafana dashboards that correlate traces, metrics, and logs in a single interface.
What You Will Learn in This Article
- Complete Jaeger setup for distributed tracing
- Prometheus configuration for metrics collection
- Grafana dashboards for unified visualization
- Loki for log collection and querying
- Correlation between traces, metrics, and logs in Grafana
- Deploying the complete stack with Docker Compose
The Complete Stack with Docker Compose
The fastest way to start the entire observability stack is with Docker Compose. The following configuration deploys all necessary components with connections already configured:
# docker-compose.observability.yaml
version: "3.9"
services:
# --- OpenTelemetry Collector ---
otel-collector:
image: otel/opentelemetry-collector-contrib:0.96.0
command: ["--config=/etc/otel/config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel/config.yaml
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "8889:8889" # Prometheus metrics export
- "13133:13133" # Health check
depends_on:
- jaeger
- loki
# --- Jaeger: Distributed Tracing Backend ---
jaeger:
image: jaegertracing/all-in-one:1.54
environment:
COLLECTOR_OTLP_ENABLED: "true"
SPAN_STORAGE_TYPE: "badger"
BADGER_EPHEMERAL: "false"
BADGER_DIRECTORY_VALUE: "/badger/data"
BADGER_DIRECTORY_KEY: "/badger/key"
volumes:
- jaeger_data:/badger
ports:
- "16686:16686" # Jaeger UI
- "14250:14250" # gRPC collector
# --- Prometheus: Metrics Backend ---
prometheus:
image: prom/prometheus:v2.50.0
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=30d"
- "--web.enable-remote-write-receiver"
- "--enable-feature=exemplar-storage"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus_data:/prometheus
ports:
- "9090:9090"
# --- Loki: Log Aggregation ---
loki:
image: grafana/loki:2.9.4
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki-config.yaml:/etc/loki/local-config.yaml
- loki_data:/loki
ports:
- "3100:3100"
# --- Grafana: Visualization ---
grafana:
image: grafana/grafana:10.3.1
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: observability
GF_FEATURE_TOGGLES_ENABLE: "traceqlEditor tempoSearch tempoBackendSearch"
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning
- grafana_data:/var/lib/grafana
ports:
- "3000:3000"
depends_on:
- prometheus
- jaeger
- loki
volumes:
jaeger_data:
prometheus_data:
loki_data:
grafana_data:
Jaeger: Distributed Tracing
Jaeger is the most widely used open source backend for distributed tracing. Originally developed by Uber and now a CNCF project, Jaeger receives traces from the Collector, stores them, and provides a UI for searching, visualizing, and analyzing distributed traces.
Integration with OpenTelemetry is native: Jaeger accepts OTLP data directly, eliminating the need for conversions. The Jaeger UI offers a waterfall view for individual traces, trace comparison, and search by service, operation, duration, and tags.
Key Jaeger Features
Trace Search: search traces by service, operation, min/max duration, tags,
and time range. Allows quickly finding problematic requests.
Waterfall View: detailed visualization of each span in the trace, with
times, attributes, events, and parent-child relationships.
Trace Comparison: side-by-side comparison of two traces to identify behavioral
differences between a fast and slow request.
Service Dependencies: automatic service dependency map, derived from
collected traces.
Prometheus: Metrics Collection
Prometheus is the standard open source monitoring and alerting system for metrics. It uses a pull model (scraping) to collect metrics from HTTP endpoints, supports a powerful query language (PromQL), and integrates natively with Grafana for visualization.
# prometheus.yml - Prometheus Configuration
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "/etc/prometheus/rules/*.yml"
scrape_configs:
# Scraping OTel Collector (exported metrics)
- job_name: "otel-collector"
static_configs:
- targets: ["otel-collector:8889"]
scrape_interval: 10s
# Collector internal metrics
- job_name: "otel-collector-internal"
static_configs:
- targets: ["otel-collector:8888"]
# Scraping applications with /metrics endpoint
- job_name: "applications"
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: (.+)
replacement: "