09 - eBPF Instrumentation - Low-Overhead Observability Without Code Changes

eBPF Instrumentation: Kernel-Level Observability

eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that allows running sandboxed programs within the kernel, without modifying kernel source code or loading kernel modules. Applied to observability, eBPF enables distributed traces, performance metrics, and network visibility without modifying application code and without agents.

While OpenTelemetry auto-instrumentation operates at the application level (intercepting library calls), eBPF operates at the kernel level, intercepting syscalls, network packets, and file system operations. This approach offers lower overhead and broader coverage, including services that cannot be traditionally instrumented.

How eBPF Works

eBPF allows loading small programs into the Linux kernel that execute in response to specific events: syscalls, network packet arrival, process context switches, file system operations. These programs are verified by the kernel to ensure safety (no crashes, no infinite loops) and are compiled into efficient bytecode.

eBPF Probe Types

eBPF programs can be attached to different kernel points, each useful for a different aspect of observability:

// Conceptual example: eBPF probe to capture TCP connections
// This program attaches to the connect() syscall

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct connection_event {
    __u32 pid;
    __u32 tid;
    __u64 timestamp_ns;
    __u32 src_addr;
    __u32 dst_addr;
    __u16 dst_port;
    char comm[16];  // process name
};

// Buffer to send events to user-space
struct {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
    __uint(key_size, sizeof(int));
    __uint(value_size, sizeof(int));
} events SEC(".maps");

// Probe attached to tcp_connect
SEC("kprobe/tcp_connect")
int trace_tcp_connect(struct pt_regs *ctx) {
    struct connection_event event = {};

    event.pid = bpf_get_current_pid_tgid() >> 32;
    event.tid = bpf_get_current_pid_tgid();
    event.timestamp_ns = bpf_ktime_get_ns();
    bpf_get_current_comm(&event.comm, sizeof(event.comm));

    // Extract destination address and port
    // ... (access kernel structures)

    // Send event to user-space
    bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,
                          &event, sizeof(event));
    return 0;
}
      

Pixie: Kubernetes Observability with eBPF

Pixie (now part of the CNCF project) is a Kubernetes observability platform that uses eBPF to provide automatic visibility into applications, networking, and infrastructure without requiring code changes, application agents, or sidecars.

# Install Pixie on a Kubernetes cluster
# Prerequisites: Kubernetes 1.21+, Linux kernel 4.14+

# 1. Install Pixie CLI
bash -c "$(curl -fsSL https://withpixie.ai/install.sh)"

# 2. Deploy Pixie to the cluster
px deploy

# 3. Verify status
px get viziers

# 4. Run queries from terminal
# View HTTP connections between services
px run px/http_data

# Latency per endpoint
px run px/service_stats

# DNS traffic
px run px/dns_data

# CPU profiling (flame graph)
px run px/perf_flamegraph -- --pod=order-service
      

Pixie automatically captures HTTP, gRPC, MySQL, PostgreSQL, Redis, Kafka, and DNS traffic at the kernel level, reconstructing requests and calculating latencies without any application instrumentation.

Cilium and Hubble: Network Observability

Cilium is a CNCF project that uses eBPF for networking and security in Kubernetes. Hubble, Cilium's observability component, provides complete visibility into network traffic between pods, including L3/L4/L7 metrics, network policies, and automatic service maps.

# Install Cilium with Hubble enabled
# helm install cilium cilium/cilium --version 1.15.0

# Cilium configuration with Hubble
apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
data:
  # Enable Hubble for observability
  enable-hubble: "true"
  hubble-listen-address: ":4244"
  hubble-metrics-server: ":9965"

  # L7 HTTP metrics
  hubble-metrics: >-
    dns
    drop
    tcp
    flow
    icmp
    http
    httpV2:sourceContext=workload-name|reserved-identity;destinationContext=workload-name|reserved-identity

  # Export to OpenTelemetry
  hubble-export-file-max-size-mb: "10"
  hubble-export-file-max-backups: "5"

  # Enable relay for UI
  hubble-relay-enabled: "true"
  hubble-ui-enabled: "true"
      

# Hubble commands for network observability

# Observe traffic in real time
hubble observe --namespace ecommerce

# Filter by HTTP protocol
hubble observe --namespace ecommerce --protocol http

# View only errors (4xx, 5xx)
hubble observe --namespace ecommerce --http-status 400-599

# Traffic between specific services
hubble observe --from-pod ecommerce/order-service \
               --to-pod ecommerce/payment-service

# Metrics for Prometheus
# Hubble metrics are exposed on :9965/metrics
# and can be scraped by Prometheus
      

eBPF vs Auto-Instrumentation: Comparison

eBPF and auto-instrumentation are complementary approaches, not alternatives. Each has specific strengths and limitations that make them suitable for different scenarios.

Current eBPF Limitations

Despite its revolutionary potential, eBPF has important limitations that affect adoption:

• Linux only: eBPF is a Linux kernel technology, not available on Windows or macOS (in production)
• Kernel version: requires kernel 4.14+ (5.8+ for advanced features), limiting legacy systems
• TLS traffic: eBPF sees data after decryption only with uprobes on SSL libraries, with additional complexity
• No business context: eBPF sees packets and syscalls, not order IDs or customer tiers
• Development complexity: writing custom eBPF programs requires kernel knowledge

Conclusions and Next Steps

eBPF represents the future of low-overhead observability. The ability to obtain distributed traces, network metrics, and profiling without modifying application code is revolutionary, especially for Kubernetes clusters with hundreds of services.

The optimal approach combines eBPF (for network and infrastructure visibility) with OpenTelemetry (for application and business context). Tools like Pixie and Cilium/Hubble make eBPF accessible even to teams without kernel experience.

In the next article, we will explore sampling strategies, analyzing how to reduce telemetry volume and associated costs without losing visibility into critical events.

Related deep dive: 08 - Context Propagation - Correlating Logs, Traces, and Metrics.

Related deep dive: 10 - Sampling Strategies - Reducing Telemetry Volume and Costs.