115 - ML-Ops: manual deploy or CI/CD, when the automatic pipeline does not pay off

You have a service that works, a model with a precise version and a way of putting them into production. One question remains that closes the module, and that no tool answers for you: should the release start by itself at every change, or does a person launch it? The fashionable answer is "automate everything".

The honest answer is that it depends on how much a mistake costs, on how many people release and on how often. In this lesson you separate two things that often travel together, automatic verification and automatic release, reproduce in miniature a classic mistake of release scripts, write a script that stops, verifies and goes back, and build yourself a criterion for deciding how far to push the pipeline in your project.

All the shell listings in this lesson were run twice with Bash 5.3.9, in a temporary folder outside the repository, with the same output (only the name of the log file created by mktemp changes).

The two GitHub Actions files were checked for YAML syntax and compared with the documentation, but they were not run by GitHub: they are schemas to adapt. The curl command against a real service was not run, and I say so where it appears.

The terms, in one paragraph

A pipeline is a sequence of automatic steps (test, build, publish) linked in a chain, like an assembly line in which each station receives the work of the previous one. CI (continuous integration) is the part that at every change runs tests and checks: it is the gatekeeper at the entrance of the main branch.

CD has two meanings that are best not confused: in continuous delivery the code is always ready to be released but the decision to release stays with a person, in continuous deployment every change that passes the checks goes into production without human intervention.

The latter is also what is called "push-to-deploy". The exit code is the number with which every program tells the system how it went: zero is "good", any other value is "problem". The pipe (|) connects the output of one command to the input of the next, and set -e is the Bash setting that stops the script at the first failed command, like a circuit breaker.

A phantom release is a deploy that seems to have succeeded, with no errors and with the service answering, but has not published the new code: the name is mine, the phenomenon is the one you will reproduce shortly.

The analogy: the autopilot and the pilot

An aircraft's autopilot is an extraordinary tool in cruise: constant, tireless, precise. But nobody entrusts it with take-off and landing without a pilot watching, because those are the moments when a mistake costs everything and conditions change. The release is the take-off of your software.

An automation that works in 99% of cases is excellent until it meets the case nobody had foreseen, and then it carries out the wrong thing with the same precision, at machine speed. This is not an argument against automation: it is an argument for knowing what replaces the human eye when you remove it.

What CI does well, without irony

Before discussing the limits, acknowledge the merits. A pipeline does three things well that a person does badly:

• It repeats identically: same sequence, same commands, same environment, every time. A person gets distracted and skips a step on Friday evening.
• It does not forget the checks: the tests always run, even when "it's only a small change".
• It scales with people: with twenty developers releasing several times a day, the manual release becomes the bottleneck and the source of incidents.

The first two entries apply to everyone, even to those who work alone, and are almost free to obtain: they are the reason why CI is the first sensible step for almost every project.

The third depends on the context: the benefit grows with the number of people and the frequency of releases, and with one person and rare releases it almost does not appear. The order of the entries does not matter, but their nature does: two are properties of the machine, one is a consequence of the organisation.

This is a minimal example of CI alone with GitHub Actions: at every push and every pull request it installs the environment exactly from the lockfile (lesson 05) and runs the fast tests (lessons 06 and 07). It goes in .github/workflows/ci.yml. The schema follows the UV guide for GitHub Actions, and is a schema to adapt: I checked it as YAML (two lines of verification with PyYAML) but GitHub did not run it.

name: ci
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
      - run: uv sync --locked
      - run: uv run pytest -q -m "not slow"

The line on: [push, pull_request] says when to start. runs-on: ubuntu-latest asks for a clean virtual machine, and "clean" is the point: it is the clean-kitchen test of lesson 01 repeated at every change.

uv sync --locked installs exactly the lockfile and fails if it is not up to date with respect to pyproject.toml, so a forgotten lockfile breaks the CI instead of passing silently. -m "not slow" excludes the tests marked as slow (the marker from lesson 06), which you can move to a separate or nightly job.

Note the two uses lines: I copied the format of the UV guide, which pins every action to a commit hash with the version in a comment (actions/checkout at v7.0.1, astral-sh/setup-uv at v9.0.0, versions consulted on 2026-11-13).

It is not pedantry: the list of tags of the astral-sh/setup-uv repository has no v9 label that follows the updates, only v9.0.0, and an old example with @v5 would point to a superseded version. A reference to a hash does not change under your feet, the same principle as the digest in the article on registries and tags.

The result is a green or red mark next to every commit. It is pure value and low risk: if something breaks you have one more failed test, not one less service. With the protected branches of lesson 03 you can also prevent the merge if the mark is red. For a complete pipeline, with image building and publishing, there is the series on ML pipelines with CI/CD, GitHub Actions and Docker; here what is needed is the criterion for deciding how far to push it.

A classic mistake: the build that fails and the release that starts anyway

The riskiest part of automation is not the pipeline: it is the script that the pipeline, or you, launch at the end. An innocent idea is to show only the last lines of the log of a long build, connecting the build to tail with a pipe. The problem comes from a rule of the language: in the Bash manual, Pipelines section, the exit code of a pipe is that of the last command, unless the pipefail option is active.

That command is tail, which always succeeds. The manual also says, in the description of set -e, that the shell does not exit if the one that fails is a command of a pipeline that is not the last. The exit code of the build is lost, and the script proceeds with the next step.

Reproducing it in miniature

To see it you need a fake build that fails. The file prints a normal message, an error on standard error and exits with code 1, as a compiler that cannot find a module would do:

#!/usr/bin/env bash
echo "compiling..."
echo "error: missing module" >&2
exit 1

Now the wrong version of the release script, with set -e convinced it is safe:

#!/usr/bin/env bash
set -e
./build.sh 2>&1 | tail -25
echo "copying the dist to production (the OLD one)"
echo "restarting the service"

The output is the proof of the crime: both compiling... and error: missing module appear, and right after them copying the dist to production (the OLD one), restarting the service and exit=0. The error is there, printed in plain sight, and the script went on anyway. It is exactly the phantom release: the process ends with success, the service answers, and in production there is the previous version, restarted.

To see the codes of every link of a pipe, Bash has the PIPESTATUS array: false | true; echo "${PIPESTATUS[@]}" prints 1 0. The first command failed, the second did not, and the overall code is the zero of the second.

Three fixes, and a surprise

The minimal fix is set -o pipefail, which makes the pipe's code that of the last failed command. It is written at the top of the script, together with -e:

#!/usr/bin/env bash
set -eo pipefail
./build.sh 2>&1 | tail -25
echo "SHOULD NOT GET HERE"

Tried, it prints the error and exits with exit=1 without reaching the last line. It is a single line, and it makes true again what set -e promised. The second fix is not to put the command that matters in a pipe: write the log to a file and read the outcome separately, so that you have both the code and the text available. A natural attempt:

#!/usr/bin/env bash
set -e
./build.sh > build.log 2>&1
RC=$?
[ "$RC" -ne 0 ] && { tail -30 build.log; exit 1; }
echo "copying the dist"

Here is the surprise, found by trying with set -e active: the script exits with exit=1, and so far so good, but it does not print the log. The output is empty. It happens because set -e interrupts the script at the very moment ./build.sh fails, before reaching RC=$? and the tail.

The safety effect is there, the diagnosis is not: at two in the morning you find yourself with "failed" and no reason. The rule I draw from it concerns the way of verifying fixes: a check is tried in the real condition (here with set -e active), not just read, because when read it looked right.

The third form works with set -e because the command is inside the condition of an if: the manual specifies that the shell does not exit if the one that fails is part of the test of an if, and that the exit code inverted by ! is also exempt. In this way the failure becomes a branch of the program instead of a sudden exit:

#!/usr/bin/env bash
set -euo pipefail
LOG=$(mktemp)
if ! ./build.sh > "$LOG" 2>&1; then
  echo "BUILD FAILED, last lines of the log:" >&2
  tail -30 "$LOG" >&2
  exit 1
fi
echo "copying the dist"

Tried, it prints BUILD FAILED, last lines of the log: followed by the log, and exits with exit=1, without copying anything: it stops and explains. The options added with respect to the minimal fix are -u, which treats a variable that was never defined as an error (a typo in a variable name does not become a silent empty string), and pipefail as a safety net for the pipes you will add later.

One limit remains to be known: set -e has many exceptions listed in the manual (lists with && and ||, loops, functions called in a context that ignores it), so it should not be considered a total guarantee but a first net.

The second defence: verify a fact that changes

Every fix above protects from a mistake you have already imagined. The defence against the ones you have not imagined is another: after the release, verify a measurable fact that changes with the modification (the served version, the commit hash, the presence of a new string in a page), not just that the service answers.

"It answers 200" is true both with the new code and with the old one: it does not distinguish. For an ML service the natural fact is the model version exposed by the health check, as in the service with the middleware from the previous article, whose /health returns model_version.

A release script that stops, verifies and goes back

A manual release can be improved a great deal without a pipeline: it is enough that the steps are in a carefully written script, which a person launches. The script that follows gathers the defences seen: it stops the release if the build fails, publishes and restarts, verifies that the expected version is being served, and goes back to the previous one if the verification fails.

To be able to run it without a server, the real commands (build, publish, read the version) come from environment variables, and I replace them with fake commands. The variable names are Italian and are kept as in the listing: NUOVO is the new version, TENTATIVI the attempts, ATTESA the wait, PRECEDENTE the previous version, SERVITA the served one. The first lines prepare the variables:

#!/usr/bin/env bash
set -euo pipefail

NUOVO="${1:?usage: deploy.sh <expected-version>}"
BUILD_CMD="${BUILD_CMD:-./build.sh}"
DEPLOY_CMD="${DEPLOY_CMD:?the command that publishes and restarts is required}"
HEALTH_CMD="${HEALTH_CMD:?the command that prints the served version is required}"
TENTATIVI="${TENTATIVI:-5}"
ATTESA="${ATTESA:-1}"

PRECEDENTE="$($HEALTH_CMD 2>/dev/null || echo unknown)"
LOG="$(mktemp)"

set -euo pipefail gathers the three safety settings seen above. NUOVO="${1:?...}" demands the argument: without the expected version the script refuses to start and prints the usage message (that is the meaning of :?).

The variables written as ${NAME:-value} have a fallback value that the outside can replace, while those written with :? have no fallback and make the script stop if they are missing: DEPLOY_CMD and HEALTH_CMD are mandatory on purpose, because a fallback value such as true would make pass a verification that verifies nothing.

Before touching anything the script asks the service which version it is serving and keeps it in PRECEDENTE: it will be needed for the rollback, and if the service is down the answer becomes unknown. A detail that bites: the command variables are expanded without quotes because they contain a command and its arguments, so they cannot contain a pipe or paths with spaces.

Tried with HEALTH_CMD="echo 3 | cat" the result is the string 3 | cat, not 3. For a check that requires a pipe, like the one on /health, the command has to be put in a separate file (I show it further below).

echo "[1/4] build (log in $LOG)"
if ! $BUILD_CMD > "$LOG" 2>&1; then
  echo "BUILD FAILED: nothing was released. Last lines:" >&2
  tail -5 "$LOG" >&2
  exit 1
fi

echo "[2/4] publishing and restarting (was in service: $PRECEDENTE)"
$DEPLOY_CMD "$NUOVO"

The build writes to a log file, without a pipe, inside an if !, the form that stops and explains. If it fails, the script exits with code 1 printing the last lines and before publishing or restarting anything: the phantom release from before cannot happen here.

Only after a successful build is DEPLOY_CMD called with the version to release, the command that in the real world copies the files and restarts the service. If that command fails, set -e stops the script with its code, with no rollback: at that point the state of the service is unknown and an automatic step cannot guess it, so it is right that it stops and tells a person.

echo "[3/4] verification (expecting version $NUOVO)"
for i in $(seq "$TENTATIVI"); do
  SERVITA="$($HEALTH_CMD 2>/dev/null || true)"
  if [ "$SERVITA" = "$NUOVO" ]; then
    echo "[4/4] OK: version $SERVITA is being served"
    exit 0
  fi
  echo "  attempt $i/$TENTATIVI: served '$SERVITA'"
  sleep "$ATTESA"
done

Here is the heart, the verification of a fact that changes. Up to TENTATIVI times, ATTESA seconds apart, the script asks for the served version and compares it with the expected one. If it matches: success, exit 0.

The retry loop exists because a service with a heavy model is not ready immediately after the restart: the previous article showed a health check that distinguishes "alive" from "ready", and with a large model the waiting time gets much longer, so the fallback of five seconds (five attempts, one second apart) is a teaching value to be adapted to your measured startup.

if [ "$PRECEDENTE" = "unknown" ]; then
  echo "VERIFICATION FAILED and no previous version known: rollback impossible" >&2
  exit 3
fi
echo "VERIFICATION FAILED: rollback to $PRECEDENTE" >&2
$DEPLOY_CMD "$PRECEDENTE"
exit 2

If after all the attempts the version is not the expected one, the script goes back to the previous one. With an unknown previous version the rollback is impossible and it says so with a different code. Note the three exit codes: 1 means "the build failed, nothing was released", 2 "released but not verified, restored" and 3 "not verified and no version to go back to".

Whoever calls the script, a person or a pipeline, can tell the three cases apart and react differently. An honest limit: after the rollback the script does not verify the served version again, and it should.

The four tests use a state file that acts as a fake "service". The read command prints the content of the file, an attiva ("activate") command writes the requested version, and an attiva_guasto ("faulty activate") one ignores it and always leaves version 2, to simulate a restart that does not bring the new code. The file is called stato ("state"). The driver is this one:

#!/usr/bin/env bash
export HEALTH_CMD=./health.sh TENTATIVI=3 ATTESA=0

echo "--- 1. all good"
printf 1 > stato
BUILD_CMD=./build_ok.sh DEPLOY_CMD=./attiva.sh ./deploy.sh 2; echo "exit=$? state=$(cat stato)"

echo "--- 2. build failed"
printf 2 > stato
BUILD_CMD=./build.sh DEPLOY_CMD=./attiva.sh ./deploy.sh 3; echo "exit=$? state=$(cat stato)"

echo "--- 3. the restart does not bring the new version"
printf 2 > stato
BUILD_CMD=./build_ok.sh DEPLOY_CMD=./attiva_guasto.sh ./deploy.sh 3; echo "exit=$? state=$(cat stato)"

echo "--- 4. service down before the release"
rm -f stato
BUILD_CMD=./build_ok.sh DEPLOY_CMD=./attiva_guasto.sh ./deploy.sh 3; echo "exit=$? state=$(cat stato)"

I ran the driver twice with the same output, apart from the name of the log file (the "log in ..." line is abbreviated below). The result, scenario by scenario:

The comparison between the second and the third scenario is instructive: in both the service stays at version 2, but in the second the build prevented any change, in the third the release took place and only the verification discovered that it had no effect.

Without the verification the third scenario would have ended with zero. The script is not ready to be copied into production, because it does not know your service, your paths and your build command: it is a model in which every line corresponds to a mistake you can really make.

In production the command that reads the version would be a separate file, for the pipe reason seen above. This is the schema, not run because it requires a service listening; the line with python3, instead, was tried on a sample response and prints 3:

#!/usr/bin/env bash
set -euo pipefail
curl -fsS http://127.0.0.1:8000/health \
  | python3 -c 'import sys, json; print(json.load(sys.stdin)["model_version"])'

It is used by passing HEALTH_CMD=./versione_servita.sh (the file name means "served version"). It assumes that the service exposes model_version in the health check, like the application in the previous article; curl's -f makes the command fail if the response is an HTTP error, -s removes the progress meter and -S still reports errors. With pipefail active, if curl cannot connect the command fails instead of passing an empty response to Python.

Manual or automatic: how to decide

A person who releases a service on one machine is in fact making a different choice from a department with twenty developers. The reasoning that follows is a reasoning about trade-offs, not a rule: the five factors in the table say when the release pipeline pays off and when it does not.

The table does not say that manual is "better": it says that for one person and rare releases the total cost is lower, and that it changes when the numbers change. If four more people who release every day join, the calculation reverses and the right choice becomes a pipeline, with the defences seen above.

A well-done manual deploy is not improvised: the exact commands, the order, the verification and the rollback are written in a document, the runbook, which is automation for people instead of for machines.

It always has the same parts: when it is used, the preconditions (green tests, chosen version, access to the server), the commands in the order in which they must be run, the verification with the expected result, the rollback with the exact commands, the traps already met and a history of updates.

The order matters here, because the preconditions come before the commands and the rollback after the verification, which is why I write it as a path and not as a list. The most useful rule to take home is that every mistake solved once becomes a line in the runbook, so that it does not repeat: it costs fifteen minutes per incident and saves hours for whoever comes after, you included.

Three questions before automating anything

"Automating" has a good reputation, and in general deserves it. But every automation is software, and software has a cost of writing, one of maintenance and one of failure. Before writing a release pipeline it is worth answering three questions:

• Do I do it often enough? An operation repeated ten times a day repays the automation in a week; one repeated once a month perhaps never repays it, because each time you run it again the pipeline has changed under you and needs fixing.
• Do I understand it well enough to describe it without ambiguity? Automating a process that is not understood freezes its defects. If every time you do it by hand a surprise, an extra step or an exception appears, it is not ready yet.
• Can I detect a failure, even an unexpected one? Automation removes the human eye from the process. If you have not replaced it with checks (tests, verification of the fact that changes, alerts), you have only removed safety.

Three "yes" answers say to automate without hesitation. A "no" to the second or the third is the most important case: it means that the real cost of automating badly is a wrong publication that nobody decided, while the real cost of not automating is a person running a command line.

The three questions have no order, but the third is the one usually discovered too late, because an expected failure is thought of in advance and an unexpected one is not, and the phantom release of the previous section was precisely a failure of that kind.

A ladder in three steps, and what changes for a model

The choice is not binary, and it is best seen as a ladder where each step presupposes the previous one. The first is the documented manual: a runbook with exact commands and verification after the release, which is enough for one service, one person and rare releases.

The second is automatic CI with manual release: the tests run by themselves at every change, while the decision and the release command stay human, and it is the best ratio between cost and benefit for most projects.

The third is full CD: every verified change is released, with automatic rollback, gradual release and monitoring that stops it if the indicators worsen. It is justified when frequency and number of people require it, and when the maturity of tests and monitoring makes it safe. Skipping a step is the recipe for phantom releases: a CD without the tests of the second step releases faster what it cannot verify.

Compared with ordinary software, the release of a model has one more risk: the new model can get worse without giving errors. It is the silent degradation seen in lesson 01 and in the series on drift: no exception, no 500, only worse predictions.

A CD pipeline for a model therefore needs a step that ordinary software does not have, a quality gate: the promotion rule with a margin from the lesson on the model registry, evaluated on a fixed set, between "trained" and "in service".

If you do not yet know how to write that gate, do not automate the next step. On the comparison between versions in production, with samples of users, there is also the article on A/B testing ML models, and on the maturity levels of the whole the series MLOps: from experiment to production.

A middle way: automation with a human gate

Between "all by hand" and "all automatic" there is continuous delivery with approval: the pipeline does everything that is safe to do alone (tests, image building, publishing to the registry) and stops in front of the release, which starts only when a person approves it. With GitHub Actions it is expressed with an environment with required reviewers. The file below extends the CI from before with a second job. It is a schema checked as YAML, not run by GitHub. The workflow's name is translated; the job id rilascia ("releases") is an identifier and is kept:

name: release
on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
      - run: uv sync --locked
      - run: uv run pytest -q -m "not slow"

  rilascia:
    needs: test
    runs-on: ubuntu-latest
    environment: production
    steps:
      - run: echo "the real release goes here, with verification and rollback"

The keyword needs: test imposes the order: the rilascia job does not start if the tests fail (without needs the jobs of a workflow run in parallel, as the workflow syntax documentation says).

environment: production links the job to an environment named production, which you configure in the repository settings; the guide to managing environments explains that you can indicate up to six people or teams as required reviewers, that the approval of just one is enough for the job to proceed, and that self-approval can be prevented. Two clarifications that change the choice.

First: the same page (consulted on 2026-11-13) says that running a workflow that cites a nonexistent environment creates one with that name, and a newly created environment has no reviewers, so until you configure them yourself the job starts without any pause.

Second: protection rules such as required reviewers are available for public repositories on all plans, while for private or internal ones, with the Free, Pro and Team plans, they are not. Check your plan before relying on that gate. The reference to the command that the job runs, here a simple echo, is the place where the script with verification and rollback from the previous section would go.

What to do now, with a project of your own

The course ends here, but the real work begins with a project of your own. The order matters, because every practice rests on the previous ones and costs more than the previous one: start from Git and a correct .gitignore (half an hour, immediate benefit); then UV with pyproject.toml and uv.lock, from which the project is rebuilt with one command (an hour); then five tests on the things you fear breaking, that is the input data, two invariants and a threshold with a margin (half a day); then external configuration with validation at startup; then MLflow with SQLite to track experiments, and the registry when you have two models to compare; the Dockerfile only when you have to move the project to another machine or serve it to someone; a release runbook written the first time you release; and CI when more than one person touches the code or you notice you forget to run the tests.

The time estimates are mine, indicative, and not measured. It is not an obligation to do everything: the criterion from the first point to the last is to add the practice when the pain it solves is real, and to note why you added it.

What you can do now

You can tell CI, which checks at every change, from automatic release, which publishes without a decision, and you know they are different choices with different costs. You can reproduce in a few lines the phantom release of a pipe with tail, explain it with a rule of the Bash manual and close it in three ways (pipefail, log to a file with if !, never the pipe on the command that matters), including the surprise of set -e hiding the log.

You can write a script that stops the release if the build fails, verifies a fact that changes, goes back to the previous version and uses distinct exit codes, and you know where it remains incomplete. You can put a human gate in front of the release with a GitHub Actions environment and which plan limit to check first.

With the previous lesson, deployment, where the model runs and how to choose, you have the four forms of putting into service and the criterion for deciding how much to automate them. The next step is not another tool: it is choosing a project of your own and taking it, one step at a time, out of the notebook.

The same series also includes ML-Ops: from notebook to service, ML-Ops: GitHub, remotes, protected branches and pull requests and ML-Ops: UV, environments and dependencies that repeat identically.