Build the artifact once and promote it unchanged
Produce a single immutable, versioned artifact in CI and deploy that exact artifact, referenced by content digest, through every environment. Rebuilding per environment lets toolchain versions and transitive dependencies drift, so the bits you tested are not the bits you ship.
- Do: build a container image once, push it to a registry, and promote it by digest from staging to production.
- Don't: run
docker buildagain in the production deploy job from the same source "to be safe".
Keep pipeline definitions in version control
Store your pipeline as code (.github/workflows/ci.yml, .gitlab-ci.yml, Jenkinsfile) in the same repository as the application, so it is reviewed, versioned, and reproducible for any past commit. Pipelines configured only in a web UI are invisible, unreviewable, and impossible to roll back with the code.
- Do: commit the workflow file and require review for changes to it.
- Don't: edit job steps directly in the CI server's UI where the change leaves no diff.
Fail fast and order stages cheapest-first
Run quick, cheap checks (lint, unit tests, secret scan) before slow, expensive ones (integration tests, end-to-end, deploys) so developers get feedback in seconds and a trivial mistake does not consume a full pipeline run. A pipeline that runs a 30-minute suite before catching a lint error wastes time and money.
- Do: put format, lint, and unit tests in the first stage and short-circuit on failure.
- Don't: schedule the longest integration suite first and discover a syntax error 25 minutes in.
Never store secrets in the repository or build logs
Keep all credentials in a dedicated secrets manager or the CI provider's encrypted store and inject them as masked variables at runtime; add secret scanning to pre-commit and CI. A secret in Git history or a build log is exposed to everyone with read access and must be rotated.
- Do: read
DATABASE_PASSWORDfrom an injected secret and prefer short-lived OIDC-federated tokens. - Don't: hardcode an API key in the workflow file or
echoa token while debugging.
Make deployments reversible before you make them frequent
Adopt a deployment strategy with built-in rollback (blue-green, canary, or GitOps revert) and rehearse it so reverting is routine, not an improvised emergency. Deploying often is only safe when recovery is fast, which is why MTTR matters as much as deployment frequency.
- Do: wire automatic rollback to smoke-test and canary-metric failures.
- Don't: deploy straight to 100% of traffic with no tested path back to the previous version.
Design database changes to be backward compatible
Use the expand-then-contract pattern so old and new application versions can run simultaneously during a rollout and so a code rollback never strands the schema. Additive changes ship first and are safe to revert; destructive changes happen only after no running code references the old shape.
- Do: add a nullable column, deploy code that writes both old and new, backfill, then drop the old column in a later release.
- Don't: rename or drop a column in the same deploy that switches the code to the new name.
Treat infrastructure as code with plan-then-apply review
Define infrastructure in version-controlled IaC and require a reviewed plan (terraform plan, helm diff) before any apply, so the effect of a change is visible and approved before it touches production. Manual console changes create drift that makes environments irreproducible.
- Do: open a PR with the plan output attached and apply only after approval.
- Don't: click through the cloud console to "quickly fix" a resource that IaC manages.
Shift security checks left into the pull request
Run dependency scanning (SCA), static analysis (SAST), secret detection, and IaC scanning on every pull request so vulnerabilities are caught when they are cheapest to fix and the author still has context. Finding the same issue in a late audit costs far more and arrives without context.
- Do: fail the PR when a dependency scan finds a high-severity CVE above your threshold.
- Don't: defer all security review to a manual gate the week before release.
Make every automation step idempotent
Write deploy scripts, IaC, and configuration so that rerunning after a partial failure converges to the desired state rather than duplicating resources or erroring. Pipelines fail and retry; a non-idempotent step turns a transient failure into a corrupted environment.
- Do: use declarative apply that reconciles to desired state and tolerates re-runs.
- Don't: append-only scripts that create a second load balancer every time the job retries.
Instrument for observability, not just monitoring
Emit structured metrics, logs, and traces (ideally via OpenTelemetry) so you can both alert on known failures and investigate unforeseen ones without shipping new code. Dashboards for known problems are necessary but insufficient when a novel incident hits.
- Do: propagate a trace ID across services and record RED metrics (rate, errors, duration) per endpoint.
- Don't: rely on grepping unstructured logs across hosts during a live incident.
Define SLOs and govern releases with an error budget
Set explicit Service Level Objectives backed by Service Level Indicators, and use the resulting error budget to decide when to keep shipping features versus pause for reliability work. Without an objective number, the velocity-versus-stability argument is decided by whoever is loudest.
- Do: pause risky feature rollouts when the 30-day error budget is exhausted.
- Don't: chase 100% availability, which is impossibly expensive and not what users need.
Use ephemeral, least-privilege build runners
Run CI jobs on ephemeral runners that are destroyed after each job and grant them only the permissions they need, scoped per pipeline. Long-lived shared runners accumulate state between builds and become a high-value target whose compromise exposes every project's secrets.
- Do: use fresh ephemeral runners and short-lived, narrowly scoped credentials per job.
- Don't: reuse a persistent runner with broad admin credentials cached on disk.
Generate an SBOM and pin dependencies by hash
Produce a Software Bill of Materials per build and pin dependencies to exact versions or content hashes, so you can answer "are we affected?" within minutes of a new CVE and so a tampered or typosquatted package cannot silently slip in. Floating version ranges and no inventory turn a disclosed vulnerability into an open-ended investigation.
- Do: lock dependencies with a hash-verified lockfile and emit a CycloneDX or SPDX SBOM as a build artifact.
- Don't: depend on
latesttags or unpinned ranges that resolve differently on each build.
Keep feature flags short-lived and clean them up
Use feature flags to decouple deploy from release and to enable instant kill switches, but remove each flag once its rollout is complete. Stale flags multiply code paths, inflate test combinations, and become latent bugs nobody remembers the meaning of.
- Do: track every flag with an owner and a removal date, and delete it after full rollout.
- Don't: leave a permanent
if (flagEnabled)branch in the codebase a year after launch.