Start with a modular monolith and extract services under pressure

Begin with a single, well-modularised deployable and extract a service only when a concrete need (independent scaling, independent deploy cadence, team ownership) forces it. Premature decomposition turns cheap in-process calls into networked failure modes and imposes the full operational tax before you have the volume to justify it.

Do: ship a modular monolith with explicit module boundaries, then carve out the one module that genuinely needs to scale separately.
Don't: open a new repo and service for every entity on day one before you understand the domain or have CI/CD and observability in place.

Align service boundaries with business capabilities and bounded contexts

Cut services along Domain-Driven Design bounded contexts and business capabilities so each service owns a cohesive slice of the domain and changes for one set of reasons. Boundaries drawn along technical layers create chatty cross-service calls and changes that ripple across many services.

Do: put everything about "ordering" (its rules, data, and API) in one service.
Don't: create separate "controller", "service", and "repository" microservices for the same capability.

Give every service its own database

Each service must own its data exclusively and expose it only through its API or events; no other service reads its tables directly. A shared database silently couples services so they can no longer deploy or change schema independently, recreating a monolith with network latency.

Do: have the Orders service publish events that the Reporting service consumes into its own store.
Don't: let Reporting run SQL joins across the Orders service's tables.

Put a timeout and circuit breaker on every remote call

Every network call can hang or fail, so bound it with a timeout and protect chronically failing dependencies with a circuit breaker that fails fast and lets the dependency recover. Unbounded calls exhaust threads and let a single slow dependency cascade into a system-wide outage.

Do: wrap an outbound HTTP/gRPC client with a timeout plus a Resilience4j circuit breaker and a fallback.
Don't: call a downstream service with the client's default infinite timeout and no failure handling.

Retry only idempotent operations, with backoff and jitter

Retry transient failures with a bounded number of attempts, exponential backoff, and jitter, but only for operations that are safe to repeat. Retrying a non-idempotent operation can duplicate side effects such as charging a card twice, and synchronous retries without jitter can produce a thundering herd that worsens an outage.

Do: make a "create order" endpoint idempotent via an idempotency key, then retry safely.
Don't: blindly retry a POST that creates a record without deduplication.

Publish events reliably with the transactional outbox

To change state and emit an event atomically, write the event to an outbox table inside the same local transaction, then relay it to the broker with a separate process or change data capture. Doing the database write and the broker publish as two independent steps eventually loses or duplicates events when a crash lands between them.

Do: insert into outbox in the same transaction as the business update, then publish from the outbox.
Don't: save the row, then call the broker in application code and hope both succeed.

Make message consumers idempotent

Brokers and retries deliver messages at least once, so consumers must tolerate duplicates by detecting and ignoring messages they have already processed. A consumer that assumes exactly-once delivery will double-apply effects whenever a message is redelivered after a transient failure.

Do: record processed message IDs (or use a natural dedup key) and skip duplicates.
Don't: assume each event is delivered exactly once and apply it unconditionally.

Propagate a correlation ID and instrument distributed tracing

Generate a correlation/trace ID at the system edge and propagate it through every call and log line, instrumenting with OpenTelemetry. Without end-to-end tracing, diagnosing which hop caused a failure or latency spike across many services becomes guesswork.

Do: standardise on OpenTelemetry propagation and include trace and span IDs in structured logs.
Don't: write per-service logs with no shared request identifier.

Evolve APIs in a backward-compatible way

Within a major version, only add optional fields and new endpoints; reserve removals, renames, and stricter validation for a new major version run in parallel during migration. Breaking a published contract in place forces every consumer to redeploy in lock-step, destroying the independent deployability that justifies microservices.

Do: add a new optional field and a v2 endpoint while keeping v1 until consumers migrate.
Don't: rename a JSON field in the existing version and break every client at once.

Verify integration with consumer-driven contract tests

Use consumer-driven contracts so each consumer declares what it needs and the provider verifies it in CI, catching breaking changes before deploy without slow, brittle end-to-end suites. Relying solely on full end-to-end tests across many services yields flaky pipelines and slow feedback.

Do: maintain Pact contracts and fail the provider build if a consumer's expectations are not met.
Don't: depend on a fragile staging environment running all services to catch contract breaks.

Design for graceful degradation and bulkheading

Decide in advance how each service behaves when a dependency is unavailable, returning a degraded but useful response where possible, and isolate resources so one struggling dependency cannot consume all of them. Systems that treat every dependency as always-available fail hard and totally instead of degrading.

Do: serve cached or default recommendations when the recommender service is down, using a separate thread pool for it.
Don't: let the product page return a 500 because an optional sidebar service timed out on the shared pool.

Externalise configuration and keep services stateless

Read configuration and secrets from the environment or a config/secret service, and keep service instances stateless so any instance can handle any request and can be scaled or replaced freely. Baking config into images or holding session state in memory blocks horizontal scaling and safe, disposable instances.

Do: inject config via environment variables and store session state in a shared cache or token.
Don't: hardcode endpoints and credentials in the image or rely on sticky sessions for in-memory state.

Give each service its own deployment pipeline

Each service should build, test, and deploy through its own automated pipeline so it can release on its own cadence without coordinating a big-bang deploy. A single shared pipeline that releases all services together is the hallmark of a distributed monolith and removes the main benefit of the architecture.

Do: deploy the Payments service independently behind health checks and a canary, several times a day if needed.
Don't: require every service to be versioned and released together in one orchestrated event.

Secure service-to-service traffic with zero-trust and mTLS

Treat the internal network as hostile: authenticate and encrypt service-to-service calls with mutual TLS, authorise based on propagated identity rather than network location, and keep secrets in a manager. Assuming everything inside the perimeter is trusted lets one compromised service move freely across the system.

Do: enforce mTLS and identity-based authorization (often via a service mesh), propagating user identity in signed tokens.
Don't: allow any in-cluster service to call any other unauthenticated over plaintext.

Expose liveness and readiness health checks

Provide distinct liveness and readiness probes so the orchestrator restarts genuinely hung instances and routes traffic only to instances that are actually ready (dependencies connected, caches warm). Conflating the two causes either traffic sent to not-yet-ready instances or needless restarts of instances that are merely warming up.

Do: return readiness false until the database connection and required dependencies are confirmed.
Don't: report healthy as soon as the process starts, before it can actually serve requests.