Best practices

Use multi-stage builds to keep runtime images small

Compile and assemble in a build stage that has the SDKs and tools, then copy only the resulting artefacts into a minimal final stage, so compilers and build dependencies never ship to production. Smaller images pull faster, start faster, and present a smaller attack surface.

  • Do: a first stage FROM golang:1.22 AS build that produces a binary, then FROM gcr.io/distroless/static with COPY --from=build /app/server /server.
  • Don't: ship the full SDK image with source, package caches, and build tools baked into the running container.

Run containers as a non-root user

A process that runs as root inside the container is root with respect to the kernel, so a container escape becomes a host compromise. Create and switch to an unprivileged user in the image, and enforce it in Kubernetes so a misbuilt image cannot run privileged.

  • Do: USER 10001 in the Dockerfile and securityContext: { runAsNonRoot: true } on the Pod.
  • Don't: leave the default root user and mount the Docker socket or grant privileged: true "to make it work".

Pin images by digest, not by a mutable tag

Tags like latest or 1 can be repointed to different content over time, so a deploy that "worked yesterday" may pull a different image today, breaking reproducibility and supply-chain integrity. Pin to the immutable SHA-256 digest for anything you deploy.

  • Do: image: registry/app@sha256:... in production manifests.
  • Don't: rely on image: app:latest and assume the running bits never change.

Order Dockerfile layers from least to most frequently changing

Each instruction is a cached layer, and a change invalidates that layer and everything after it. Install dependencies (which change rarely) before copying application source (which changes constantly) so rebuilds reuse the dependency layers.

  • Do: COPY package.json package-lock.json ./ then RUN npm ci, and only afterwards COPY . ..
  • Don't: COPY . . first, which busts the dependency-install cache on every source edit.

Set both resource requests and limits

Requests let the scheduler place Pods sensibly and define their Quality of Service; limits stop a single container from starving its neighbours. Without requests, Pods land on overcommitted nodes and are evicted first; without limits, a leak can take down a node.

  • Do: set memory request = limit for predictable, Guaranteed-class workloads, and a CPU request with a sensible limit.
  • Don't: omit both and hope, then debug noisy-neighbour outages and surprise OOM kills.

Define liveness and readiness probes correctly

Readiness gates traffic so requests never hit a Pod that is still starting or temporarily overloaded; liveness restarts a genuinely stuck container. Conflating them, or making liveness too aggressive, causes needless restart loops that look like instability.

  • Do: a cheap readiness probe on a /readyz endpoint and a more conservative liveness probe on /healthz.
  • Don't: point a tight liveness probe at a slow startup path so healthy Pods get killed mid-boot.

Never store secrets in images, ConfigMaps, or Git

ConfigMaps and image layers are not confidential, and Kubernetes Secrets are only base64-encoded at rest unless you enable encryption. Treat secret material as sensitive end to end and keep it out of build artefacts and version control.

  • Do: enable etcd encryption at rest, restrict Secrets with RBAC, and inject via mounted Secrets or an external secrets operator.
  • Don't: bake an API key into an ENV layer or commit a Secret manifest with real values to the repository.

Apply default-deny NetworkPolicies

By default every Pod can reach every other Pod, so one compromised workload can pivot freely. Start from a default-deny policy per namespace and explicitly allow only the ingress and egress each workload needs (least privilege).

  • Do: a default-deny-all NetworkPolicy plus narrow allow rules from the specific client Pods, enforced by a capable CNI (Calico, Cilium).
  • Don't: leave the namespace fully open and rely on Services being "internal" as if that were a security boundary.

Manage state with the right controller and real storage

Stateless workloads belong in Deployments; workloads that need stable identity and per-replica durable storage belong in StatefulSets with PersistentVolumeClaims. Writing important data to a container's ephemeral writable layer guarantees data loss on reschedule.

  • Do: a StatefulSet with a volumeClaimTemplate for a database, and a backed-up StorageClass.
  • Don't: run a database as a Deployment writing to local container storage and discover the loss after a node drains.

Use a PodDisruptionBudget for availability during disruptions

Voluntary disruptions (node drains, cluster upgrades, autoscaler scale-down) can evict several replicas at once and cause an outage. A PodDisruptionBudget caps how many replicas may be unavailable simultaneously so maintenance stays safe.

  • Do: set minAvailable: 2 (or a percentage) on a multi-replica Deployment that must stay serving.
  • Don't: run two replicas with no budget and have a single node drain take both down at once.

Make deployments declarative and version-controlled (GitOps)

Imperative kubectl edit changes drift from any record and are hard to review or roll back. Declare desired state in Git and let a reconciler (Argo CD, Flux) apply and continuously enforce it, turning rollbacks into reverts and changes into reviewable pull requests.

  • Do: store manifests/Helm values in Git and have an in-cluster agent reconcile the live cluster to them.
  • Don't: hot-patch production by hand and lose the audit trail and the ability to reproduce the state.

Scan images and pin a minimal base in CI

Vulnerabilities arrive through base images and transitive packages, so an image that was clean at build can be exploitable later. Scan on every build, rebuild on base-image updates, and prefer minimal/distroless bases that contain fewer packages to patch.

  • Do: run an image scanner (Trivy, Grype) in the pipeline and fail the build on fixable critical CVEs.
  • Don't: build once from a fat base and never re-scan or rebuild, accumulating unpatched CVEs.

Handle termination signals and set sensible grace periods

Kubernetes sends SIGTERM and waits terminationGracePeriodSeconds before SIGKILL; a process that ignores SIGTERM drops in-flight requests on every rollout. Trap the signal, stop accepting new work, drain, and exit cleanly, and use a preStop hook plus readiness flip to coordinate with load balancers.

  • Do: handle SIGTERM to finish in-flight requests, and use a short preStop sleep so endpoints update before the process exits.
  • Don't: run the app as PID 1 in shell form so it never receives signals, then see 502s on every deploy.