Define all infrastructure as version-controlled code
Provision every resource through declarative Infrastructure-as-Code kept in version control, never by clicking through a console, so environments are reproducible, reviewable, and auditable. Manual changes cause drift that no one can reconstruct after an incident. Do: declare a VPC, subnets, and instances in a Terraform module reviewed via pull request and applied by a pipeline. Don't: hand-create a production database in the console and try to remember the settings later.
Treat servers as immutable and replace rather than patch
Build a new image and roll out fresh instances to change a running system, instead of mutating live servers in place, so you eliminate configuration drift and make rollback a redeploy of the previous image. In-place patching produces snowflake servers that no two of which are alike. Do: bake a new golden image or container image, then do a rolling replace. Don't: SSH into production to edit config files and install packages on the fly.
Spread workloads across at least two availability zones
Distribute instances, replicas, and database standbys across two or more availability zones so the loss of a single data centre does not take down your service. A single-AZ deployment is a single point of failure no matter how many instances it has. Do: run the application tier and a database standby in separate AZs behind a load balancer. Don't: put every replica in one zone because it is marginally cheaper or simpler.
Keep backend resources in private subnets
Place application servers, databases, and internal services in private subnets with no public IP and no inbound internet route, exposing only a load balancer or gateway publicly and using a NAT gateway for outbound traffic. Directly internet-reachable backends dramatically widen the attack surface. Do: put the database in a private subnet reachable only from the app tier's security group. Don't: assign a public IP to a database "just to connect to it quickly".
Apply least-privilege IAM everywhere
Grant each identity only the permissions it actually needs, scoped to specific resources, and prefer short-lived workload identities over long-lived static keys. Broad wildcard permissions turn any single compromise into a full account breach. Do: give a service a role allowing read on one specific bucket. Don't: attach an administrator policy to an application "to make it work for now".
Store secrets in a secrets manager, never in code or images
Keep credentials, tokens, and keys in a dedicated secrets manager and inject them at runtime, with rotation and audited access, rather than embedding them in source, container images, or IaC files. Secrets committed to a repository or baked into an image leak widely and are hard to revoke. Do: read a database password from a secrets manager via the instance role at startup. Don't: hardcode an API key in a Dockerfile or commit it to Terraform.
Right-size resources and continuously manage cost
Match instance sizes to real utilisation, delete idle resources, commit to reserved capacity for steady baseline load, use spot for interruptible work, and tag everything so spend is attributable. Cost is an architectural property, and untracked cloud spend grows silently. Do: tag resources by team and act on right-sizing recommendations each month. Don't: leave a fleet of oversized, untagged instances running because nobody owns the bill.
Design for failure with health checks and automatic failover
Assume every component will eventually fail and build redundancy, health checks that remove unhealthy targets, and automatic failover so the system degrades gracefully rather than going dark. Hardware, zones, and dependencies fail; hoping they won't is not a strategy. Do: configure load-balancer health checks and a standby database that promotes automatically. Don't: rely on a single instance and a manual restart when it crashes at night.
Test backups and disaster-recovery procedures regularly
Verify that backups actually restore and that failover runbooks work by rehearsing them on a schedule, with explicit recovery time and recovery point objectives. An untested backup is a hope, not a recovery plan, and most failover surprises surface only during a real drill. Do: run a quarterly restore drill into an isolated environment and measure the recovery time. Don't: assume nightly snapshots are usable because the backup job reports success.
Use auto scaling with sensible bounds and stateless workloads
Scale capacity automatically against load signals, but set minimum and maximum bounds, warm-up, and cooldown to avoid thrashing, and keep the scaled tier stateless so any instance can serve any request. Without bounds, a traffic spike or a metric loop can scale into a runaway bill or oscillation. Do: auto scale a stateless web tier on request rate with a defined max, storing session state externally. Don't: auto scale a stateful service that pins users to specific instances.
Make changes idempotent and operations retry-safe
Ensure that provisioning runs and API operations produce the same end state when repeated, and make create-like operations safe to retry with idempotency keys, because cloud calls time out and pipelines re-run. Non-idempotent operations corrupt state on retry and make automation dangerous. Do: rely on declarative IaC convergence and pass an idempotency key on payment-like POSTs. Don't: write provisioning scripts that fail or duplicate resources if executed twice.
Roll out releases progressively and keep rollback fast
Ship new versions with canary or blue-green strategies that limit blast radius and let you revert quickly, gated on health and latency metrics rather than a single big-bang switch. A release that cannot be rolled back fast turns a small defect into a prolonged outage. Do: send 5% of traffic to a canary, watch the error rate, then ramp or roll back automatically. Don't: replace 100% of production at once with no automated rollback path.
Instrument everything with logs, metrics, and traces
Build observability in from the start - structured logs, metrics, and distributed traces - so you can diagnose novel failures in distributed systems by asking new questions, not just watching pre-built dashboards. Systems you cannot observe become systems you cannot operate. Do: emit correlated request IDs across services and alert on SLO-based metrics. Don't: discover outages only when customers complain because nothing is instrumented.
Encrypt data in transit and at rest by default
Enable TLS for all network traffic and encryption at rest for storage and databases as a baseline, terminating TLS at the edge and using managed keys, so data exposure from interception or stolen media is prevented. Unencrypted data is a compliance and breach risk with little upside. Do: enforce HTTPS, enable storage encryption, and use mutual TLS between internal services. Don't: leave an internal service on plaintext because "it's only on the private network".