Glossary

Observability

Observability is the degree to which the internal state of a system can be inferred from its external outputs - primarily telemetry such as logs, metrics, and traces. A system is observable when you can answer new questions about its behaviour without shipping new code to emit more data. The term is borrowed from control theory, where observability measures how well internal states are reconstructable from outputs.

Telemetry

Telemetry is the data a system emits about its own behaviour, collected and transmitted to a backend for analysis. In modern practice it is grouped into signals: logs, metrics, traces, and increasingly continuous profiles. OpenTelemetry standardises how this telemetry is generated, described, and exported.

Metric

A metric is a numeric measurement captured over time, such as request rate, error count, or latency. Metrics are cheap to store and aggregate because they are pre-aggregated numbers with low cardinality dimensions, which makes them ideal for dashboards and alerting. The classic metric types are counters, gauges, and histograms.

Log

A log is a timestamped, typically immutable record of a discrete event, ranging from unstructured text lines to structured JSON. Logs carry high-detail context that is hard to pre-aggregate, which makes them excellent for debugging specific events but expensive at high volume. Structured logging attaches key-value fields so logs can be queried and correlated.

Trace

A trace represents the end-to-end journey of a single request as it flows through a distributed system, composed of one or more spans. It shows causal and timing relationships between operations across service boundaries, making it the primary tool for understanding latency in microservices. Each trace is identified by a unique trace ID propagated across services.

Span

A span is the basic unit of a trace: a single named, timed operation such as an HTTP handler, a database query, or an RPC call. Spans carry a start time, duration, attributes, events, status, and references to a parent span, forming a tree or directed graph within a trace. The first span in a trace is the root span.

Distributed Tracing

Distributed tracing is the practice of tracking a request across all the services it touches by propagating a shared context (trace ID and span ID) between them. It reconstructs the full call graph so engineers can find which service or operation contributed to latency or errors. W3C Trace Context is the standard header format for propagating this context over HTTP.

Cardinality

Cardinality is the number of unique values a dimension or label can take, and in metrics it is the number of unique time series produced by all label combinations. High cardinality (for example a label holding user IDs) multiplies storage and memory cost dramatically and is the most common cause of metrics-backend overload. Choosing low-cardinality labels is a core skill in instrumentation.

Counter

A counter is a cumulative metric that only ever increases (or resets to zero on restart), used for things like total requests served or total bytes sent. You almost always read a counter through a rate function rather than its raw value, because the absolute total is rarely meaningful on its own. Resets are detected and handled by query engines such as PromQL.

Gauge

A gauge is a metric that can go up or down, representing a value sampled at a point in time, such as current memory usage, queue depth, or active connections. Unlike a counter, a gauge's instantaneous value is directly meaningful. Gauges are typically aggregated with functions like average, min, max, or sum.

Histogram

A histogram samples observations (such as request durations) into configurable buckets and also exposes a count and sum, enabling latency distribution analysis and quantile estimation. Prometheus histograms expose cumulative bucket counters that are aggregatable across instances before computing quantiles. Native (exponential) histograms reduce the need to pre-pick bucket boundaries.

Quantile

A quantile is a cut point that divides a distribution so a given fraction of observations fall below it; the 0.99 quantile (p99) is the value under which 99% of observations lie. Quantiles describe tail behaviour that averages hide, which is why latency SLOs are usually expressed at p95 or p99. With Prometheus histograms, quantiles are estimated from buckets using the histogram_quantile function.

SLI (Service Level Indicator)

An SLI is a carefully defined quantitative measure of some aspect of the level of service provided, such as the ratio of successful requests to total requests. Good SLIs are expressed as a ratio of good events to valid events, which keeps them between 0 and 100%. SLIs are the raw measurements that SLOs set targets against.

SLO (Service Level Objective)

An SLO is a target value or range for an SLI over a time window, for example "99.9% of requests succeed over 30 days". SLOs translate reliability into a number the whole organisation can reason about and trade off against feature velocity. They are internal targets, typically stricter than any externally promised SLA.

SLA (Service Level Agreement)

An SLA is a contract with users or customers that specifies the consequences (often financial) of failing to meet a level of service. SLAs are business agreements, whereas SLOs are the internal engineering targets set deliberately tighter so problems are caught before the SLA is breached. Not every service needs an SLA, but most benefit from SLOs.

Error Budget

An error budget is the amount of unreliability an SLO permits - for a 99.9% availability SLO it is the remaining 0.1% of allowed failures over the window. It reframes reliability as a budget that can be spent on releases and experiments: when the budget is exhausted, the team prioritises stability over new features. Error budgets create a shared, data-driven decision rule between development and operations.

Burn Rate

Burn rate is how fast a service is consuming its error budget relative to the rate that would exactly exhaust it over the SLO window. A burn rate of 1 means the budget will last exactly the window; a burn rate of 10 means it will be gone in a tenth of the time. Multi-window, multi-burn-rate alerts page fast for severe burns and slowly for gradual ones, balancing precision and recall.

Golden Signals

The four golden signals from Google SRE are latency, traffic, errors, and saturation - the minimum set of metrics recommended for monitoring a user-facing service. Latency measures how long requests take, traffic measures demand, errors measure failed requests, and saturation measures how full the most constrained resource is. Instrumenting these four gives broad coverage with little effort.

RED Method

The RED method monitors a service by three metrics: Rate (requests per second), Errors (failed requests per second), and Duration (latency distribution). It is a request-centric framing well suited to instrumenting microservices and request-driven workloads. RED complements USE, which is resource-centric.

USE Method

The USE method, coined by Brendan Gregg, characterises each resource by Utilization, Saturation, and Errors. It is designed for diagnosing systems and hardware bottlenecks (CPU, memory, disk, network) by asking, for every resource, how busy it is, how much extra work is queued, and whether it is producing errors. USE is resource-centric and pairs naturally with the request-centric RED method.

OpenTelemetry (OTel)

OpenTelemetry is a CNCF project and vendor-neutral standard that provides APIs, SDKs, and tools to generate, collect, and export logs, metrics, and traces. It defines a common data model and wire protocol so instrumentation is decoupled from any specific backend, letting you switch vendors without re-instrumenting. It is the result of merging the earlier OpenTracing and OpenCensus projects.

OTLP (OpenTelemetry Protocol)

OTLP is the OpenTelemetry Protocol, the standard wire format for transmitting telemetry between SDKs, the Collector, and backends. It is defined in Protocol Buffers and runs over gRPC or HTTP, carrying traces, metrics, and logs in a single unified format. Most observability backends now accept OTLP natively.

OpenTelemetry Collector

The OpenTelemetry Collector is a vendor-agnostic standalone agent or gateway that receives, processes, and exports telemetry. Its pipeline is built from receivers, processors, and exporters, letting you batch, filter, sample, enrich, and fan out data without changing application code. It decouples applications from backends and is the recommended place to centralise telemetry policy.

Instrumentation

Instrumentation is the act of adding code or configuration that makes a system emit telemetry. It is manual when you write spans and metrics in your code, and automatic when an agent or library injects telemetry into known frameworks without code changes. Good instrumentation balances coverage against cost and cardinality.

Context Propagation

Context propagation is the mechanism that carries trace context (and baggage) across process and service boundaries so spans from different services join into one trace. Over HTTP this is typically done with the W3C traceparent and tracestate headers. Without propagation, each service produces disconnected traces and the end-to-end picture is lost.

W3C Trace Context

W3C Trace Context is a standard that defines the traceparent and tracestate HTTP headers for propagating trace identifiers across services in an interoperable way. traceparent carries the version, trace ID, parent span ID, and trace flags; tracestate carries vendor-specific key-value data. Standardising the headers lets tools from different vendors share a single trace.

Sampling

Sampling is the practice of recording only a subset of traces to control cost and volume while preserving representative signal. Head-based sampling decides at the start of a trace, before its outcome is known; tail-based sampling decides after the trace completes, so it can keep all errors and slow requests. The right strategy trades cost against the chance of dropping the traces you most need.

Head-Based Sampling

Head-based sampling makes the keep-or-drop decision at the root of a trace, before the request's outcome is known, and propagates that decision to all downstream spans. It is simple and cheap and guarantees complete traces, but it cannot preferentially keep errors or slow requests because the decision happens too early. It is often called probabilistic sampling when based on a fixed ratio.

Tail-Based Sampling

Tail-based sampling buffers all spans of a trace until it completes, then applies rules to decide which traces to keep, such as always keeping traces with errors or latency above a threshold. It captures the most interesting traces but requires buffering complete traces (usually in the Collector), which costs memory and complicates horizontally scaled deployments. It is the standard way to keep rare but important traces affordably.

Exemplar

An exemplar is a reference attached to a metric data point (typically a histogram bucket) that links it to a specific trace, usually via a trace ID. Exemplars let you jump from an aggregate metric - for example a spike in p99 latency - directly to an example trace that contributed to it. They bridge the metrics and traces signals for faster root-cause analysis.

Structured Logging

Structured logging emits log events as machine-parsable key-value records (commonly JSON) instead of free-form text. Structure lets you filter, aggregate, and correlate logs reliably - for example by trace_id or tenant - without brittle regular expressions. It is a prerequisite for treating logs as queryable data rather than a wall of text.

Log Level

A log level classifies the severity or importance of a log event, with common levels being TRACE, DEBUG, INFO, WARN, and ERROR. Levels let operators filter verbosity dynamically, emitting detail in development and only warnings and errors in production. Choosing the right level for each event keeps logs signal-rich and affordable.

Aggregation

Aggregation is the process of combining many telemetry data points into summary values, such as summing request counts across instances or averaging CPU usage across a fleet. It reduces data volume and reveals system-level behaviour that individual data points hide. Care is needed because some operations (such as averaging percentiles) are statistically invalid.

PromQL

PromQL is Prometheus's functional query language for selecting and aggregating time-series data, supporting rate calculations, aggregation operators, and quantile estimation. It operates on instant vectors and range vectors and powers both dashboards and alerting rules. Its rate() and histogram_quantile() functions are central to most reliability queries.

Scrape

A scrape is Prometheus's pull-based collection of metrics, in which the server periodically issues an HTTP request to a target's /metrics endpoint and parses the exposition. The scrape interval determines time-series resolution and storage cost. Pull-based scraping gives the monitoring system control over collection timing and makes target health (up/down) directly observable.

Pull vs Push

Pull-based collection has the monitoring system fetch metrics from targets (as Prometheus scrapes /metrics), while push-based collection has applications send telemetry to a collector or gateway (as OTLP push does). Pull simplifies discovery and liveness detection; push suits short-lived jobs and serverless functions that may not be reachable when scraped. Many stacks combine both, using a push gateway or the OTel Collector.

Alerting

Alerting is the automated detection of conditions that need human attention, evaluating rules over telemetry and notifying responders when thresholds or SLO burn rates are crossed. Good alerts are actionable, urgent, and tied to user-facing symptoms rather than internal causes. Alert fatigue from noisy, non-actionable alerts is a leading cause of missed real incidents.

Dashboard

A dashboard is a curated visual arrangement of charts and panels that present telemetry for monitoring and exploration. Effective dashboards answer a specific question (such as "is this service healthy?") and lead from high-level signals to detail. Dashboards complement, but do not replace, alerting and ad-hoc querying.

Cardinality Explosion

A cardinality explosion is an uncontrolled growth in the number of unique time series caused by attaching high-cardinality values (such as user IDs, request IDs, or full URLs) to metric labels. It can exhaust the memory and storage of a metrics backend and is a frequent cause of outages in the monitoring system itself. The fix is to move high-cardinality data to logs or traces and keep metric labels low-cardinality.

MTTR (Mean Time To Recovery)

MTTR is the average time taken to restore service after an incident begins, a core measure of operational resilience. Observability reduces MTTR by shortening the detect-and-diagnose phases, letting responders find the failing component quickly. It is often paired with MTTD (mean time to detect) and MTBF (mean time between failures).

Service Map

A service map (or service graph) is a visualisation of the services in a system and the call relationships between them, usually derived automatically from trace data. It shows dependencies, request rates, and error rates between services, making architecture and failure propagation visible. It is a common entry point for navigating from a symptom to the responsible service.

Baggage

Baggage in OpenTelemetry is a set of user-defined key-value pairs propagated alongside trace context across service boundaries, available to all downstream spans. It carries cross-cutting business context - such as a tenant or experiment ID - so it can be added to spans or used for routing decisions. Baggage is propagated to all downstream services, so it must avoid sensitive data and excessive size.

Continuous Profiling

Continuous profiling is the always-on collection of low-overhead performance profiles (such as CPU and memory) from production, increasingly treated as a fourth observability signal. It pinpoints which functions or lines consume resources, complementing traces that show where time goes between services. OpenTelemetry has standardised a profiling signal and OTLP profiles format.

Resource (OpenTelemetry)

In OpenTelemetry a Resource is an immutable set of attributes describing the entity producing telemetry, such as service.name, service.version, and deployment.environment. Resource attributes are attached to all signals from that entity, enabling consistent correlation of logs, metrics, and traces back to a single service. Semantic conventions define standard resource attribute names.

Semantic Conventions

Semantic conventions are OpenTelemetry's standardised names and meanings for attributes (such as http.request.method or db.system) so telemetry from different sources is consistent and portable. Following them lets backends and dashboards understand data without per-source configuration. They are the shared vocabulary that makes vendor-neutral observability practical.

Retention

Retention is how long telemetry is stored before being deleted or downsampled, traded off against storage cost and the need to investigate past incidents. High-resolution recent data is usually kept short-term, while aggregated metrics may be downsampled and kept for capacity planning. Different signals warrant different retention because their cost and value profiles differ.

Correlation

Correlation is linking related telemetry across signals - for example jumping from a metric spike to the traces and logs from the same time and service. It is enabled by shared identifiers such as trace_id, consistent resource attributes, and exemplars. Strong correlation is what turns three separate signals into a single investigative workflow.