Instrument with OpenTelemetry, not vendor SDKs
Instrument your code against the vendor-neutral OpenTelemetry API so the telemetry you emit is decoupled from any single backend. This means you can switch or add backends - or send to several at once - by changing exporter configuration rather than re-instrumenting application code. Do export via OTLP through the Collector; don't scatter a proprietary vendor agent's API calls throughout your business logic, because that locks you in and makes migration a rewrite.
Keep metric labels low-cardinality
Treat every metric label as a multiplier on your time-series count, and only use dimensions with a small, bounded set of values such as HTTP method, status class, or route template. Do label requests by route="/users/{id}"; don't label by the concrete URL /users/12345, by raw user ID, or by request ID, because each unique value creates a new series and an unbounded label can crash the metrics backend. Push high-cardinality detail into logs or trace attributes instead.
Define SLOs from user-facing symptoms
Base your reliability targets on indicators the user actually experiences - successful request ratio, end-to-end latency - rather than internal causes like CPU or pod restarts. Do express an SLI as good events over valid events and set an SLO with an explicit window (for example 99.9% over 30 days); don't set an arbitrary "five nines" target everywhere, because each extra nine is far more expensive and an unrealistically strict SLO generates constant, ignorable alerts.
Alert on symptoms and SLO burn rate, not causes
Page humans for things that are both urgent and user-affecting, ideally driven by how fast you are burning the error budget. Do use multi-window, multi-burn-rate alerts so a severe burn pages immediately and a slow burn pages gently; don't alert on every internal metric crossing a static threshold (high CPU, a single failed request), because cause-based alerts are noisy, often non-actionable, and train responders to ignore the pager.
Use structured logging with correlation IDs
Emit logs as structured key-value records (typically JSON) and include the active trace_id and span_id on every line. Do log {"level":"error","trace_id":"...","order_id":"...","msg":"payment declined"} so you can filter and pivot to the trace; don't write free-form strings like Error processing order 42 that require brittle regex to parse and cannot be correlated to a trace. Structure is what turns logs from a wall of text into queryable data.
Sample traces deliberately, keeping the interesting ones
Decide a sampling strategy on purpose rather than recording everything or nothing. Do use tail-based sampling to keep all error traces and slow requests while dropping a large fraction of routine successes, so cost stays bounded but the diagnostically valuable traces survive; don't rely solely on a low fixed head-sample rate, because it discards errors at the same rate as successes and you lose exactly the traces you need during an incident.
Standardise on semantic conventions and resource attributes
Use OpenTelemetry semantic conventions for attribute names (http.request.method, db.system) and set consistent resource attributes (service.name, service.version, deployment.environment) on every signal. Do adopt the standard names so backends and shared dashboards work without per-service configuration; don't invent your own attribute keys per team, because inconsistent naming breaks cross-service correlation and forces bespoke parsing for every source.
Centralise telemetry policy in the Collector
Run an OpenTelemetry Collector (as a per-node agent and/or a gateway) and put batching, retries, redaction, sampling, and routing there. Do enforce PII scrubbing and sampling once in the Collector pipeline so every service inherits the policy; don't bake backend endpoints, sampling ratios, and redaction rules into each application, because changing policy would then require redeploying every service.
Scrub sensitive data before it leaves the process
Treat telemetry as a data-exfiltration risk and strip secrets and PII early. Do redact tokens, passwords, full payment details, and personal data in the SDK or Collector before export, and avoid putting them in span attributes, log fields, or metric labels; don't log raw request bodies or authorization headers "for debugging", because telemetry is widely accessible and often retained, turning a convenience into a compliance breach.
Set retention and downsampling per signal
Match how long you keep each signal to its cost and investigative value rather than applying one blanket retention. Do keep high-resolution metrics for a short window and downsample to coarser aggregates for long-term trends, keep error traces longer than routine ones, and tier log retention by importance; don't keep every raw signal at full fidelity forever, because storage cost grows without bound and most old high-resolution data is never queried.
Propagate context across every boundary, including async
Ensure trace context flows through synchronous calls and asynchronous hops alike so traces stay connected end to end. Do propagate the W3C traceparent header on outbound HTTP and carry context through message queues and thread pools using the SDK's context mechanisms; don't assume auto-instrumentation covers every path - custom thread pools, batch jobs, and some messaging clients silently drop context and split one request into several disconnected traces.
Make telemetry actionable with runbooks and ownership
Pair every alert with a clear owner and a runbook describing likely causes and first steps. Do link the alert to a dashboard scoped to the affected service and to a documented response procedure; don't fire an alert into a shared channel with no owner and no guidance, because an alert nobody owns and nobody knows how to act on does not reduce time to recovery and erodes trust in the system.
Test your observability like you test your code
Verify that instrumentation actually works before you depend on it in an incident. Do assert in tests that key spans and metrics are emitted with the expected attributes, run game-days that exercise alerts and dashboards, and add a dead-man's-switch heartbeat to detect silent telemetry failures; don't discover during an outage that a metric was never wired up, a label was misspelled, or an alert never fired - broken observability is worse than none because it gives false confidence.