What is the difference between monitoring and observability?
Monitoring is the act of collecting and alerting on a predefined set of metrics and conditions you already know to watch - it answers known questions like "is CPU above 90%?". Observability is the broader property of being able to ask new, unanticipated questions about a system from its telemetry without shipping new instrumentation. In practice monitoring is a capability you build on top of an observable system: you need rich, high-context telemetry (logs, metrics, traces) before you can both watch known failure modes and investigate novel ones.
What are the three pillars of observability?
The three pillars are logs, metrics, and traces, each answering a different question. Metrics tell you that something is wrong and how much, with cheap aggregatable numbers ideal for dashboards and alerts; logs tell you what happened in a specific event with high-detail context; traces tell you where in a distributed call path the problem occurred and how long each step took. The pillars are most powerful when correlated - for example jumping from a latency-metric spike to an example trace and then to that request's logs.
Do I still need logs if I have metrics and traces?
Yes - the three signals are complementary, not interchangeable. Metrics are pre-aggregated numbers that show trends and trigger alerts cheaply, and traces show the structure and timing of a request across services, but neither captures the arbitrary high-detail context of a discrete event the way a log does. Logs remain the best place for verbose, high-cardinality, ad-hoc information such as a full error message, a payload snippet, or a business decision that does not fit a metric label or span attribute.
How do I choose between metrics, logs, and traces for a given piece of data?
Use metrics for low-cardinality numeric values you aggregate and alert on (request rate, error ratio, queue depth), use traces for understanding latency and causality across services, and use logs for high-cardinality or verbose detail tied to a specific event. The deciding factor is usually cardinality and aggregation: anything with many unique values (user IDs, request IDs, full URLs) belongs in logs or trace attributes, never in metric labels, where it would explode the time-series count.
What is OpenTelemetry and why should I use it?
OpenTelemetry is a CNCF vendor-neutral standard providing APIs, SDKs, and the Collector to generate and export traces, metrics, and logs in a common format (OTLP). Its main benefit is decoupling: you instrument your code once against a stable, open API, and you can change observability backends - or send to several at once - without re-instrumenting. It is the result of merging OpenTracing and OpenCensus and is now the de facto standard, supported by virtually every modern backend.
What does the OpenTelemetry Collector do and do I need one?
The Collector is a standalone process that receives telemetry, processes it through a configurable pipeline of receivers, processors, and exporters, and forwards it to one or more backends. It lets you centralise policy - batching, retry, redaction of sensitive fields, sampling, attribute enrichment, and fan-out - without changing application code or redeploying services to switch backends. You do not strictly need one for a small system, but running a Collector as an agent (per node) or gateway (centralised) is the recommended production pattern.
How do I reduce the cost of my observability stack?
Cost is driven mostly by data volume and cardinality, so the highest-leverage levers are sampling traces, dropping or aggregating low-value metrics and logs, and removing high-cardinality labels. Use tail-based sampling to keep errors and slow traces while discarding routine ones, downsample old metrics, set retention per signal based on investigative value, and move high-cardinality fields out of metrics into logs or trace attributes. Doing this filtering centrally in the Collector keeps the policy in one place.
What is the difference between head-based and tail-based sampling?
Head-based sampling decides whether to keep a trace at its start, before the outcome is known, then propagates that decision downstream; it is cheap and produces complete traces but cannot preferentially keep errors or slow requests. Tail-based sampling buffers all spans of a trace until it finishes, then applies rules such as "keep all errors and anything over 1 second", capturing the interesting traces at the cost of memory and a more complex Collector deployment. Many teams combine a low head-sample rate with tail rules for errors and high latency.
What are SLIs, SLOs, and error budgets?
An SLI is a measured quality indicator, usually a ratio of good events to valid events, such as successful requests over total requests. An SLO is a target for that indicator over a window, for example 99.9% over 30 days. The error budget is the allowed shortfall - the remaining 0.1% - which the team can spend on releases and risk; when it runs low, the rule is to slow down and prioritise reliability over new features, giving development and operations a shared, data-driven decision.
How many nines do I actually need?
It depends on the cost of downtime and user expectations, not on aiming as high as possible - each additional nine is dramatically more expensive. As a reference, 99.9% allows about 43 minutes of downtime per month, 99.99% about 4.3 minutes, and 99.999% about 26 seconds. Set the SLO from real user impact and business cost, because over-provisioning reliability wastes engineering effort that could go to features, and an SLO that is too strict generates alert fatigue.
Why does high cardinality break my metrics backend?
Each unique combination of metric name and label values creates a separate time series that the backend must index and keep in memory, so attaching a high-cardinality value like a user ID or request ID multiplies the series count without bound. This "cardinality explosion" can exhaust memory and storage and take down the monitoring system itself - ironically causing an outage in the tool meant to detect outages. Keep metric labels low-cardinality and push high-cardinality detail into logs or trace attributes instead.
What metrics should I instrument first for a new service?
Start with the golden signals - latency, traffic, errors, and saturation - or equivalently the RED method (Rate, Errors, Duration) for request-driven services. These give broad coverage of user-facing health with minimal effort: you can tell whether the service is up, how busy it is, how often it fails, and how slow it is. Add the USE method (Utilization, Saturation, Errors) per resource when you need to diagnose infrastructure bottlenecks underneath the service.
How do I correlate logs, metrics, and traces?
Correlation relies on shared identifiers and consistent metadata: inject the active trace_id and span_id into every log line, use the same OpenTelemetry resource attributes (service.name, service.version, deployment.environment) across all signals, and attach exemplars to histograms so a metric data point links to an example trace. With these in place you can pivot from a latency spike on a dashboard to a representative trace, and from that trace to the exact log lines for the request, in a single workflow.
What makes a good alert?
A good alert is actionable, urgent, and tied to a user-facing symptom rather than an internal cause, so that paging a human is justified and the responder knows roughly what to do. Alerting on SLO burn rate - using multi-window, multi-burn-rate rules that page fast for severe budget burn and slowly for gradual burn - gives high precision and recall while avoiding noise. Alerts that are not actionable should be downgraded to dashboards or tickets, because alert fatigue causes responders to miss the real incidents.
What is context propagation and why does my trace break across services?
Context propagation is the passing of trace context - the trace ID and current span ID - from one service to the next, typically via the W3C traceparent HTTP header, so downstream spans attach to the same trace. A trace breaks into disconnected pieces when a hop fails to propagate this header: a client that does not inject it, a proxy that strips it, or an async boundary (a queue or thread pool) that loses the context. The fix is to ensure every framework, client, and messaging layer in the path reads and writes the standard headers.
Should I use a pull or push model to collect metrics?
Pull-based collection (Prometheus scraping a /metrics endpoint) lets the monitoring system control timing and detect target liveness directly, and it simplifies service discovery for long-lived services. Push-based collection (sending OTLP to a Collector or push gateway) suits short-lived jobs, serverless functions, and clients behind NAT that cannot be scraped reliably. Many production stacks use both - scraping long-lived services while pushing telemetry from ephemeral workloads - often unifying them through the OpenTelemetry Collector.
Is OpenTelemetry tracing stable and ready for production?
OpenTelemetry's tracing specification, API, and the SDKs for major languages reached a stable (1.0) state, making tracing production-ready and safe to depend on for instrumentation that will not break under the stability guarantees. Metrics also reached stability, while logs and the newer profiling signal have matured later and may still be evolving in some languages. Check the component and language-specific status pages before relying on a signal, but tracing and metrics are widely used in production today.