11 - Production Deployment: Docker, Scaling and Monitoring

The problem: an agent does not behave like a web service

A web service answers in a few tenths of a second, remembers nothing between one request and the next and, if you switch it off, loses at most the requests in flight. An agent does the opposite on all three points: a task lasts seconds or minutes because inside it there is a loop of calls to the model and to the tools, it carries state with it (what it has already done, what it still lacks) and, if the process is stopped halfway, it may already have performed an action with side effects. Putting an agent in a container and giving it three replicas makes it run, but it does not make it manageable.

This article takes those three points one at a time: how to stop a process that is working without losing the task, how to tell "it is starting" from "it is stuck" from "it cannot serve right now", how to decide how many replicas you need when the CPU tells you nothing, and what to measure and log to notice trouble. Before starting, a statement about the state of the tests: the Python listings run locally with fake, scripted models and agents, twice with identical output. The Dockerfile and the Kubernetes manifests are instead schemas to adapt, not executed: there is no Docker and no cluster on this machine, and the manifests were checked only as YAML syntax (two valid documents), not applied.

This article comes after those on the agent and before the one on costs: orchestration has already introduced tracers and queues, evaluation tells you how to know whether the agent works, and security how to limit the damage it can do. Here we talk about keeping it up.

The container: a few lines that matter

An image for a Python agent is nothing special, with a single exception: the way the process is started. The Dockerfile below uses two stages, because compilers are needed to install certain packages and are not needed to run them; the final stage copies only what was installed. The agent user without a login shell keeps the process from running as an administrator inside the container.

# --- stage 1: build ---
FROM python:3.12-slim AS builder
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
    && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

# --- stage 2: run ---
FROM python:3.12-slim AS runtime
WORKDIR /app
COPY --from=builder /install /usr/local
COPY src/ ./src/
RUN useradd --create-home --shell /usr/sbin/nologin agent
USER agent
ENV PYTHONUNBUFFERED=1
EXPOSE 8080
# exec form: python is process 1 and receives SIGTERM directly
ENTRYPOINT ["python", "-m", "src.agent_server"]

The line that matters is the last one. Docker's documentation says that the shell form of ENTRYPOINT starts the program as a child of /bin/sh -c, which does not forward signals: the program is not process 1 and does not receive the SIGTERM of docker stop. With the exec form (the list in square brackets) it is Python itself that receives the signal, and it can react as in the listing of the next section. Two common habits are worth dropping: there is no HEALTHCHECK, because the probes are declared by the Kubernetes manifest (it exists for plain Docker, with default interval and timeout of 30 seconds), and no key in the image, because an image gets copied, published and forgotten.

When you do not need all this: for an agent that handles few requests on a single server, a container (or even a systemd unit) with automatic restart covers the case. Kubernetes solves the problems of many replicas and many nodes; if you do not have them, it brings along a cluster to maintain.

Stopping well: SIGTERM and half-finished tasks

Every time you update the image, the cluster moves a pod or the autoscaler reduces the replicas, the processes receive a SIGTERM. Kubernetes' documentation describes the sequence: the kubelet asks the runtime to stop the containers by sending TERM to the main process, with a default grace period of 30 seconds (terminationGracePeriodSeconds); once that expires, the process is killed. A web service gets by because requests are short. An agent in the middle of a two-minute task does not.

The listing shows the minimal solution. The signal handler interrupts nothing: it only raises a flag, and the loop reads it between one task and the next. The state lives in a SQLite table (in the manifest it would be Redis or PostgreSQL, outside the pod), so that pod B can resume from task 3 without anyone telling it. The signal is sent by the program to itself after the second task, to reproduce without a cluster what the kubelet would do; the agent is a function that returns a string, and no model is called.

import os
import signal
import sqlite3
import tempfile

stop = False


def on_termination(number, frame):
    global stop
    stop = True  # the loop reads it between one task and the next


signal.signal(signal.SIGTERM, on_termination)


def fake_agent(task):
    return f"answer to {task}"


def run(db, name, send_sigterm_after=None):
    done = 0
    while not stop:
        row = db.execute(
            "SELECT id, text FROM tasks WHERE status = 'queued' ORDER BY id LIMIT 1"
        ).fetchone()
        if row is None:
            print(f"{name}: queue empty")
            return
        id_, text = row
        outcome = fake_agent(text)
        db.execute(
            "UPDATE tasks SET status = 'done', outcome = ? WHERE id = ?", (outcome, id_)
        )
        db.commit()
        done += 1
        print(f"{name}: task {id_} completed")
        if send_sigterm_after == done:
            os.kill(os.getpid(), signal.SIGTERM)  # as the kubelet would
    print(f"{name}: SIGTERM received, exiting after {done} tasks")


with tempfile.TemporaryDirectory() as folder:
    db = sqlite3.connect(os.path.join(folder, "state.db"))
    db.execute("CREATE TABLE tasks (id INTEGER PRIMARY KEY, text TEXT, status TEXT, outcome TEXT)")
    for n in range(1, 6):
        db.execute("INSERT INTO tasks (text, status) VALUES (?, 'queued')", (f"question {n}",))
    db.commit()

    run(db, "pod A", send_sigterm_after=2)
    stop = False  # pod B is a new process: we simulate it here
    run(db, "pod B")
    print(db.execute("SELECT status, COUNT(*) FROM tasks GROUP BY status").fetchall())

# Output:
# pod A: task 1 completed
# pod A: task 2 completed
# pod A: SIGTERM received, exiting after 2 tasks
# pod B: task 3 completed
# pod B: task 4 completed
# pod B: task 5 completed
# pod B: queue empty
# [('done', 5)]

Two things to read in the output. First, no task was lost or executed twice: pod A finished the task in progress and then exited. Second, this works because every task lasts less than the grace period. If a task can last longer, you have three choices: raise terminationGracePeriodSeconds (as in the manifest further on), split the task into steps with a resume point after each, or accept that it will be repeated. In the latter case the tools with side effects must tolerate repetition, that is, be idempotent: this is discussed in the article on advanced tool calling. A SIGKILL, which no code can intercept, arrives anyway if the period expires or if the node dies, so the state table remains necessary even with an orderly shutdown.

Three probes, three different questions

Kubernetes offers three probes, and the most frequent confusion is using one to answer all three questions. Each has a different consequence when it fails, and that is why each must be fed by a different check.

• Startup: "has it finished starting?". If it fails beyond the threshold, the kubelet kills the container and applies the restart policy.
• Liveness: "is it still making progress?". It serves to detect situations such as a deadlock; if it fails beyond the tolerance, the container is restarted.
• Readiness: "does it make sense to send it traffic now?". If it fails the pod leaves the Service's endpoints, but is not restarted.

The practical consequence lies in the third item and the second: if the liveness probe checked Redis, a Redis outage would make all the replicas restart together, and on restart they would still find Redis down, in a loop that makes the incident worse. Redis down is a readiness problem (do not send me tasks) and not a liveness one (I am not the fault). The listing separates the three questions into three pure functions; the real checks (a ping to Redis, the last turn of the main loop) are yours to supply, here they are four hand-written states.

from dataclasses import dataclass


@dataclass
class State:
    last_loop_turn: float   # seconds since the last turn of the main loop
    started: bool           # initial loading finished
    redis_ok: bool
    key_present: bool


LOOP_LIMIT_S = 120  # example choice: a turn longer than the longest expected task


def live(s):
    """200 if the process is making progress. Does not look at dependencies."""
    return (200, "alive") if s.last_loop_turn < LOOP_LIMIT_S else (503, "loop stuck")


def ready(s):
    """200 if it makes sense to send it traffic now."""
    if not s.started:
        return 503, "starting"
    if not s.redis_ok:
        return 503, "redis unreachable"
    if not s.key_present:
        return 503, "model key missing"
    return 200, "ready"


def started(s):
    return (200, "started") if s.started else (503, "loading in progress")


cases = {
    "all fine":          State(3, True, True, True),
    "redis is down":     State(3, True, False, True),
    "still starting":    State(0, False, False, True),
    "loop stuck":        State(400, True, True, True),
}
for name, s in cases.items():
    print(f"{name:17} live={live(s)[0]} ready={ready(s)[0]} startup={started(s)[0]}  ({ready(s)[1]})")

# Output:
# all fine          live=200 ready=200 startup=200  (ready)
# redis is down     live=200 ready=503 startup=200  (redis unreachable)
# still starting    live=200 ready=503 startup=503  (starting)
# loop stuck        live=503 ready=200 startup=200  (ready)

In the output, the "redis is down" row keeps the replica alive (200) but takes it out of traffic (503); the "loop stuck" row is the only one that asks for a restart. Note that in the fourth case readiness still says 200: a stuck process keeps looking ready until liveness stops it, and that is a good reason to keep the loop threshold close to the duration of the longest task (here 120 seconds, an example value, not advice). Another choice in this schema: the probe reads a state already computed in the background, instead of opening a connection on every request, so as not to turn the probe into a load.

In the manifest, the startup probe is the one that protects slow starts. Kubernetes' documentation suggests giving it the same check as liveness with a failureThreshold × periodSeconds product long enough to cover the worst case of startup: here 18 × 10 seconds, that is up to 180 seconds, chosen as an example for an agent that loads indexes or documents at the start. The rest of the manifest sets the CPU and memory requests (the CPU limit is omitted: it is my choice, debatable, so as not to slow down an already slow loop) and takes the model key from a Secret.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent
spec:
  replicas: 3
  selector:
    matchLabels: {app: agent}
  template:
    metadata:
      labels: {app: agent}
    spec:
      terminationGracePeriodSeconds: 120   # longer than the longest task you agree to finish
      containers:
        - name: agent
          image: registry.example.com/agent:1.0.0
          ports:
            - {name: http, containerPort: 8080}
          resources:
            requests: {cpu: 500m, memory: 1Gi}
            limits: {memory: 2Gi}
          env:
            - name: MODEL_KEY
              valueFrom:
                secretKeyRef: {name: agent-secrets, key: model-key}
          startupProbe:
            httpGet: {path: /health/startup, port: http}
            periodSeconds: 10
            failureThreshold: 18          # 18 x 10 s = up to 180 s to start
          livenessProbe:
            httpGet: {path: /health/live, port: http}
            periodSeconds: 15
            failureThreshold: 3
          readinessProbe:
            httpGet: {path: /health/ready, port: http}
            periodSeconds: 10
            failureThreshold: 3

The Secret is not encrypted by itself: the documentation warns that by default Kubernetes stores it unencrypted in the cluster's datastore (etcd), and whoever has access to the API, or can create a pod in the same namespace, can read it. The base64 value you see in manifests is an encoding, not a protection. For this reason the key is neither in the image nor in the repository: it is created in the cluster (for example with kubectl create secret from a local file) and encryption at rest is enabled if you manage the cluster yourself.

How many replicas: the autoscaler's formula

For an agent the CPU is a poor load signal: the process spends almost all its time waiting for the model. A pod can have ten tasks in progress and the CPU at 5%. The horizontal autoscaler (HPA) accepts custom metrics: the number of active tasks per pod, or the length of the queue. Before writing the manifest it is worth knowing what it does with them, because the rule is simple: the documentation writes it as desiredReplicas = ceil[currentReplicas × currentMetricValue / desiredMetricValue], and skips any action when the ratio is close enough to 1.0, with a default tolerance of 0.1.

The listing reproduces only that formula, the tolerance and the minimum and maximum limits: it is a reduced version, because the real controller also considers pods that are not ready, missing metrics and scaling policies. It serves to answer the questions you ask before configuring anything: how many replicas will it ask for? Does it stabilize?

import math

TOLERANCE = 0.1  # controller default (Kubernetes documentation)


def desired_replicas(current, current_metric, target_metric, minimum, maximum):
    ratio = current_metric / target_metric
    if abs(ratio - 1.0) <= TOLERANCE:
        return current  # close enough to 1: no action
    desired = math.ceil(current * ratio)
    return max(minimum, min(maximum, desired))


# per-pod metric: active tasks, target 5 per pod
print("active tasks per pod, target 5")
for current, per_pod in [(3, 5.2), (3, 8), (3, 2), (10, 12), (10, 60)]:
    d = desired_replicas(current, per_pod, 5, minimum=2, maximum=20)
    print(f"  {current:2} replicas, {per_pod:4} per pod -> {d}")

# external metric: queue depth, target 10 tasks per replica.
# With AverageValue the per-pod metric is total / current replicas.
print("queue with 120 tasks, target 10 per replica")
for current in (3, 8, 12, 20):
    d = desired_replicas(current, 120 / current, 10, minimum=2, maximum=20)
    print(f"  {current:2} replicas -> {d}")

# Output:
# active tasks per pod, target 5
#    3 replicas,  5.2 per pod -> 3
#    3 replicas,    8 per pod -> 5
#    3 replicas,    2 per pod -> 2
#   10 replicas,   12 per pod -> 20
#   10 replicas,   60 per pod -> 20
# queue with 120 tasks, target 10 per replica
#    3 replicas -> 12
#    8 replicas -> 12
#   12 replicas -> 12
#   20 replicas -> 12

Three readings. With 3 replicas and 5.2 tasks per pod the HPA does nothing (ratio 1.04, within the tolerance), while at 8 it goes to 5. With 10 replicas and 60 tasks per pod the formula would ask for 120 replicas and the maximum limit stops it at 20: at that point the queue grows and no autoscaler saves you, so the maximum is also a decision about cost and about the load on downstream services. Finally, with the queue of 120 tasks and a target of 10 per replica the system converges to 12 whatever the starting point, and that is precisely why it is better to measure the queue and not the CPU.

The HPA manifest uses the first metric, active tasks per pod, with a stabilization window of 300 seconds on the way down: the documentation explains that the HPA looks at the desired states computed over that interval and uses the highest, so as not to remove pods because of a short pause. On the way up there is no wait and at most 4 pods per minute are added (example values). A custom metric does not exist on its own: you need a component that makes it available to the cluster's metrics API, and it is one more piece to install and keep up to date.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: agent
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: agent}
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric: {name: agent_active_tasks}
        target: {type: AverageValue, averageValue: "5"}
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - {type: Pods, value: 4, periodSeconds: 60}
    scaleDown:
      stabilizationWindowSeconds: 300

If the agent processes tasks taken from a queue, there is an alternative: KEDA, a separate component that, as its documentation says, can bring the workload down to zero replicas when there are no messages and reactivate it when one arrives, driving the HPA for the rest. Scaling to zero costs a cold start at the first task, so it suits deferred jobs and not someone waiting for a reply in a chat. The HPA too can reach zero replicas with external or object metrics, not with CPU alone, according to its documentation.

What to measure, and where not to hurt yourself with labels

Infrastructure metrics (CPU, memory, restarts) tell you the pod is alive, not that the agent is working well. The ones an agent needs are few and answer precise questions:

• Tasks finished by outcome: a counter with the outcome label (ok, error, expired) to compute the failure share.
• Task duration: a histogram in seconds, to see the long tail, not just the average.
• Active tasks: a gauge that goes up and down, the same metric the autoscaler reads.
• Calls to the model and to the tools, with tokens consumed: the data that will feed the cost control of the next article.

The common criterion is that each metric answers a decision: alert, scale or explain a cost. A metric nobody reads is a memory cost in the server that collects it. The listing shows the first three with the official prometheus_client library (0.26.0), in a private registry so that the output does not depend on the process.

from prometheus_client import CollectorRegistry, Counter, Gauge, Histogram, generate_latest

reg = CollectorRegistry()  # private registry: the output does not depend on the process
tasks = Counter("agent_tasks", "Finished tasks", ["outcome"], registry=reg)
duration = Histogram("agent_task_duration_seconds", "Task duration", buckets=[1, 5, 30], registry=reg)
active = Gauge("agent_active_tasks", "Tasks in progress", registry=reg)

active.inc()
for outcome, seconds in [("ok", 0.8), ("ok", 4.0), ("error", 12.0)]:
    tasks.labels(outcome=outcome).inc()
    duration.observe(seconds)
active.dec()
active.inc()

for line in generate_latest(reg).decode().splitlines():
    if not line.startswith("#") and "_created" not in line:
        print(line)

# the design mistake: a label with unbounded values
reg2 = CollectorRegistry()
wrong = Counter("agent_tasks_by_id", "no", ["task_id"], registry=reg2)
for n in range(1000):
    wrong.labels(task_id=f"t{n}").inc()
series = [r for r in generate_latest(reg2).decode().splitlines() if r.startswith("agent_tasks_by_id_total")]
print("series created with task_id as a label:", len(series))

# Output:
# agent_tasks_total{outcome="ok"} 2.0
# agent_tasks_total{outcome="error"} 1.0
# agent_task_duration_seconds_bucket{le="1.0"} 1.0
# agent_task_duration_seconds_bucket{le="5.0"} 2.0
# agent_task_duration_seconds_bucket{le="30.0"} 3.0
# agent_task_duration_seconds_bucket{le="+Inf"} 3.0
# agent_task_duration_seconds_count 3.0
# agent_task_duration_seconds_sum 16.8
# agent_active_tasks 1.0
# series created with task_id as a label: 1000

From the output you learn something the documentation confirms: the counter called agent_tasks appears as agent_tasks_total, because the library adds the suffix; the histogram buckets are cumulative (the 5-second bucket also counts the 1-second one); duration is measured in seconds, the base unit that Prometheus' guidelines recommend. The bucket thresholds (1, 5, 30) are examples and must be chosen around the duration you actually observe. The last line is the trap: using an unbounded identifier as a label creates one time series for each value, a thousand tasks, a thousand series. The same guidelines say not to use labels for high-cardinality dimensions, such as user identifiers or emails. The task identifier goes in the logs, not in the metrics.

Logs you can follow

If a task goes through twelve calls and three pods, a log line saying "error in the tool" is not enough: you must be able to filter all the lines of that task. The solution is a trace identifier present in every line, and in Python the right place to keep it is a context variable (contextvars), which stays tied to the task even in asynchronous code without having to pass the identifier from one function to the next. The JSON format then lets you search by field. The listing sets the identifier once per task and prints it on every line.

import contextvars
import json
import logging
import sys

trace_id = contextvars.ContextVar("trace_id", default="-")


class JsonFormatter(logging.Formatter):
    def format(self, record):
        entry = {
            "level": record.levelname,
            "trace_id": trace_id.get(),
            "event": record.getMessage(),
        }
        entry.update(getattr(record, "fields", {}))
        return json.dumps(entry, ensure_ascii=False, sort_keys=True)


log = logging.getLogger("agent")
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
log.addHandler(handler)
log.setLevel(logging.INFO)


def task(task_id, steps):
    trace_id.set(f"trace-{task_id}")  # a single assignment per task
    log.info("task_started", extra={"fields": {"task": task_id}})
    for n, tool in enumerate(steps, start=1):
        log.info("tool_called", extra={"fields": {"iteration": n, "tool": tool}})
    log.warning("task_finished", extra={"fields": {"iterations": len(steps)}})


task("t1", ["search", "read"])
task("t2", ["search"])

# Output:
# {"event": "task_started", "level": "INFO", "task": "t1", "trace_id": "trace-t1"}
# {"event": "tool_called", "iteration": 1, "level": "INFO", "tool": "search", "trace_id": "trace-t1"}
# {"event": "tool_called", "iteration": 2, "level": "INFO", "tool": "read", "trace_id": "trace-t1"}
# {"event": "task_finished", "iterations": 2, "level": "WARNING", "trace_id": "trace-t1"}
# {"event": "task_started", "level": "INFO", "task": "t2", "trace_id": "trace-t2"}
# {"event": "tool_called", "iteration": 1, "level": "INFO", "tool": "search", "trace_id": "trace-t2"}
# {"event": "task_finished", "iterations": 1, "level": "WARNING", "trace_id": "trace-t2"}

The ordering of the keys (sort_keys) makes the output identical on every run, handy for comparing logs in tests; the time is omitted only in this example, in real systems the aggregator or the formatter adds it. A warning: agent logs often contain users' text, prompts and responses, so decide what you record and for how long, and never write the keys. For traces with nested spans, with the hierarchy task, iteration and call, the minimal tracer in the article on orchestration shows the structure.

Frequently asked questions

What you can do now

You can write an image whose process receives SIGTERM and finishes the task in progress before exiting, keeping the state outside the pod so that another replica can resume. You can separate the three probes according to what happens when they fail, compute by hand what the autoscaler will ask for and why a maximum that is too low becomes a queue, choose a few metrics that motivate a decision and avoid the mistake of unbounded labels. And you can say where the listings end: no line was tried on a cluster, so the values in the manifests are starting points to measure against your own load.

To see a graph with a cap and a verifier put to the test, there is the constructed example of a research assistant; for the limits that no deployment solves, The Future of AI Agents: Emergent Capabilities and Limitations.