Observability9 apr 2026•Reading time15 min

07 - Backend Integration - Jaeger, Prometheus, and Grafana Setup

Jaeger (tracing backend), Prometheus (metrics), Grafana (visualization), integration, stack setup (Docker Compose, Kubernetes), example dashboards.

ObservabilityOpenTelemetryMonitoringTracing

What do you think about this article?

At a Glance

Jaeger (tracing backend), Prometheus (metrics), Grafana (visualization), integration, stack setup (Docker Compose, Kubernetes), example dashboards.

Backend Integration: Jaeger, Prometheus, and Grafana

After instrumenting applications and configuring the Collector, the next step is connecting telemetry data to storage and visualization backends. The most popular open source stack combines Jaeger for distributed traces, Prometheus for metrics, and Grafana as the unified visualization layer. Together, these tools form a complete, production-ready observability system.

In this article, we will configure the entire stack from scratch, from Docker Compose deployment to creating Grafana dashboards that correlate traces, metrics, and logs in a single interface.

What You Will Learn in This Article

  • Complete Jaeger setup for distributed tracing
  • Prometheus configuration for metrics collection
  • Grafana dashboards for unified visualization
  • Loki for log collection and querying
  • Correlation between traces, metrics, and logs in Grafana
  • Deploying the complete stack with Docker Compose

The Complete Stack with Docker Compose

The fastest way to start the entire observability stack is with Docker Compose. The following configuration deploys all necessary components with connections already configured: 


# docker-compose.observability.yaml
version: "3.9"

services:
  # --- OpenTelemetry Collector ---
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.96.0
    command: ["--config=/etc/otel/config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel/config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "8889:8889"   # Prometheus metrics export
      - "13133:13133" # Health check
    depends_on:
      - jaeger
      - loki

  # --- Jaeger: Distributed Tracing Backend ---
  jaeger:
    image: jaegertracing/all-in-one:1.54
    environment:
      COLLECTOR_OTLP_ENABLED: "true"
      SPAN_STORAGE_TYPE: "badger"
      BADGER_EPHEMERAL: "false"
      BADGER_DIRECTORY_VALUE: "/badger/data"
      BADGER_DIRECTORY_KEY: "/badger/key"
    volumes:
      - jaeger_data:/badger
    ports:
      - "16686:16686" # Jaeger UI
      - "14250:14250" # gRPC collector

  # --- Prometheus: Metrics Backend ---
  prometheus:
    image: prom/prometheus:v2.50.0
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-remote-write-receiver"
      - "--enable-feature=exemplar-storage"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"

  # --- Loki: Log Aggregation ---
  loki:
    image: grafana/loki:2.9.4
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml
      - loki_data:/loki
    ports:
      - "3100:3100"

  # --- Grafana: Visualization ---
  grafana:
    image: grafana/grafana:10.3.1
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: observability
      GF_FEATURE_TOGGLES_ENABLE: "traceqlEditor tempoSearch tempoBackendSearch"
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
      - jaeger
      - loki

volumes:
  jaeger_data:
  prometheus_data:
  loki_data:
  grafana_data:

Jaeger: Distributed Tracing

Jaeger is the most widely used open source backend for distributed tracing. Originally developed by Uber and now a CNCF project, Jaeger receives traces from the Collector, stores them, and provides a UI for searching, visualizing, and analyzing distributed traces.

Integration with OpenTelemetry is native: Jaeger accepts OTLP data directly, eliminating the need for conversions. The Jaeger UI offers a waterfall view for individual traces, trace comparison, and search by service, operation, duration, and tags.

Key Jaeger Features

Trace Search: search traces by service, operation, min/max duration, tags, and time range. Allows quickly finding problematic requests.

Waterfall View: detailed visualization of each span in the trace, with times, attributes, events, and parent-child relationships.

Trace Comparison: side-by-side comparison of two traces to identify behavioral differences between a fast and slow request.

Service Dependencies: automatic service dependency map, derived from collected traces.

Prometheus: Metrics Collection

Prometheus is the standard open source monitoring and alerting system for metrics. It uses a pull model (scraping) to collect metrics from HTTP endpoints, supports a powerful query language (PromQL), and integrates natively with Grafana for visualization. 


# prometheus.yml - Prometheus Configuration
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "/etc/prometheus/rules/*.yml"

scrape_configs:
  # Scraping OTel Collector (exported metrics)
  - job_name: "otel-collector"
    static_configs:
      - targets: ["otel-collector:8889"]
    scrape_interval: 10s

  # Collector internal metrics
  - job_name: "otel-collector-internal"
    static_configs:
      - targets: ["otel-collector:8888"]

  # Scraping applications with /metrics endpoint
  - job_name: "applications"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: "${1}:${2}"

  # Scraping Jaeger
  - job_name: "jaeger"
    static_configs:
      - targets: ["jaeger:14269"]

  # Scraping Prometheus itself
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

Grafana: Unified Visualization

Grafana is the visualization layer that unifies traces, metrics, and logs in a single interface. It natively supports Jaeger, Prometheus, and Loki as data sources, allowing you to create dashboards that correlate all telemetry signals. 


# grafana/provisioning/datasources/datasources.yml
apiVersion: 1
datasources:
  # Prometheus for metrics
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      exemplarTraceIdDestinations:
        - name: traceID
          datasourceUid: jaeger
      httpMethod: POST

  # Jaeger for traces
  - name: Jaeger
    type: jaeger
    access: proxy
    url: http://jaeger:16686
    uid: jaeger
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true
        filterBySpanID: true
      tracesToMetrics:
        datasourceUid: Prometheus

  # Loki for logs
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    uid: loki
    jsonData:
      derivedFields:
        - datasourceUid: jaeger
          matcherRegex: "trace_id=(\\w+)"
          name: TraceID
          url: "${__value.raw}"

Signal Correlation in Grafana

The true power of Grafana emerges when the three data sources are configured for correlation. This means being able to navigate from a metric to a trace, from a trace to logs, and back, in a continuous debug flow.

Correlation Flow in Grafana

Metrics dashboard: see a spike in P99 latency → click on "Exemplar" in the graph → open trace in Jaeger → identify the slow span (payment-gateway) → click "View Logs" → open filtered logs in Loki with trace_id → find the error message "connection timeout to payment provider."

This flow transforms debugging from hours to minutes.

Loki: Log Aggregation

Loki is Grafana's log aggregation system, designed to be cost-effective and simple to operate. Unlike Elasticsearch, Loki indexes only labels (metadata), not the full log content, drastically reducing storage costs and operational complexity. 


# loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  max_query_series: 500
  max_query_parallelism: 2

analytics:
  reporting_enabled: false

Essential Queries for Each Backend

  • Jaeger: search by service, operation, min/max duration, tags (e.g., error=true)
  • Prometheus (PromQL): rate(http_requests_total[5m]), histogram_quantile(0.99, ...)
  • Loki (LogQL): {service="order-service"} |= "error" | json | trace_id != ""
  • Grafana Explore: split view to compare metrics and traces side-by-side

Conclusions and Next Steps

The Jaeger + Prometheus + Loki + Grafana stack provides a complete, open source, production-ready observability system. The key to success is correlation: configuring data sources in Grafana to navigate fluidly between metrics, traces, and logs, transforming debugging from a manual, lengthy process into a guided, fast flow.

Docker Compose is ideal for development and testing, while in production the stack is deployed on Kubernetes with dedicated Helm charts (jaeger-operator, kube-prometheus-stack, loki-stack).

In the next article, we will dive into context propagation, analyzing how to correlate logs and traces through trace ID, the W3C Baggage protocol for transporting metadata between services, and patterns for effective correlation across all signals.

Related deep dive: 08 - Context Propagation - Correlating Logs, Traces, and Metrics.

Related deep dive: 09 - eBPF Instrumentation - Low-Overhead Observability Without Code Changes.

Frequently asked questions

What is the most popular open source stack for observability?

The most popular open source stack combines Jaeger for distributed traces, Prometheus for metrics, and Grafana as the unified visualization layer.

How can I quickly start the entire observability stack?

The fastest way to start the entire observability stack is with Docker Compose.

What is the main advantage of using Jaeger's native integration with OpenTelemetry?

Jaeger accepts OTLP data directly, eliminating the need for conversions.

How does Grafana enable signal correlation between traces, metrics, and logs?

Grafana enables signal correlation by allowing you to create dashboards that correlate all telemetry signals from multiple data sources.

Sources and further reading

Related Series

Deepen your knowledge with these related learning paths.

Commenti

Caricamento commenti...

Sign in to comment

Discuss this article in the community

Join the Telegram group to discuss with other developers, ask questions and share your experiences.

Blog ChannelCommunity Group

Guida completa AI Agents (ebook)Affiliate linkWhat does it mean

Did you like this article?

Explore other content on the blog or discover my projects

All ArticlesMy Projects