111 - ML-Ops: container registries, tags, digests and versions that do not change under your feet

An image built on your laptop is of little use if the server that has to run it cannot get it. A container registry is a network archive of images: you publish them (push), others download them (pull). But a registry brings with it a versioning problem that is subtler than it looks: the name you use to ask for an image is not necessarily a stable identity. In this article you see the difference between tag and digest, why latest is a trap and when a registry is really needed.

A word first on what was verified. I wrote the docker commands (push, pull, tag, save, load) from the official documentation but did not run them: in the text they are declared schema to adapt or deduced output, not executed. I did really run the hash computation with sha256sum and a miniature registry written in Python, which reproduces the behavior of tags and digests with the same rules, to show the mechanism with real numbers.

The words of the registry

A registry is the network warehouse of images: Docker Hub is the best known, and GitHub, the big clouds and company servers have their own. It is like a library: you deposit the book, others borrow it. A repository is the shelf with all the versions of one same image (churn-api), and mind you, the word is the same as in Git but means something else.

A tag is a text label attached to a version, like 0.1.1 or latest, and it can be detached and reattached elsewhere. A hash is the fingerprint of a content, computed with a mathematical function: the same content always gives the same fingerprint, and a different content, even by a single bit, gives a completely different one. The digest of an image is the hash of its manifest, the small document in the registry that lists the layers the image is made of. Finally the login requires credentials to write (and, in private registries, to read as well), while the retention is the policy that decides which old versions to keep and which to delete.

The difference between tag and digest is visible in a single drawing: tags are arrows that can be moved, digests are the contents they point to.

The values in the drawing are illustrative. In the drawing latest points, after the second push, to the same content as 0.1.1; before the second push it pointed to that of 0.1.0. The docker pull documentation confirms the starting point: if you do not specify a tag, Docker uses latest as the default.

The analogy: the phone book and the tax ID number

In your phone's contacts you write "Marco - dentist". That name is convenient, but it is a label: tomorrow it can point to a different Marco, if you change the number. A tax ID number, on the other hand, identifies one person only and is never reassigned. The tag (churn-api:0.1.0, latest) is the readable, movable label; the digest (sha256:...) is the identity computed from the very content of the image: if even a single byte changes, the digest changes.

How to read an image name

registry.example.com / churn-api : 0.1.0
 registry              repository  tag

registry.example.com / churn-api @ sha256:<fingerprint>
 registry              repository  digest

The first name reads left to right: the registry where the image lives (Docker Hub if omitted; GitHub uses ghcr.io; the clouds have their own), the repository, that is, the name of the image that gathers all its versions, and the tag, the label of one version. In the second, the digest is indicated with @ instead of the colon. The documentation of docker image pull devotes a section to pulling "by digest (immutable identifier)" and explains that an image's ID is a SHA256 digest covering configuration and layers: two different tags on the same image have the same ID and the layers are stored only once.

What a hash is: the test with three files

To understand the digest it is enough to see a hash at work. The block creates two files of eight characters that differ by a single letter (model-v1 and model-v2) and a third with the same content as the first, then computes their SHA-256 fingerprint, showing only the first 16 hexadecimal characters of the 64 digits:

printf 'model-v1' > f1.bin
printf 'model-v2' > f2.bin
printf 'model-v1' > f3.bin
sha256sum f1.bin f2.bin f3.bin | cut -c1-16,65-
# 1a1f4502024df8a6  f1.bin
# c13b1cdc9ebe20ea  f2.bin
# 1a1f4502024df8a6  f3.bin

Run twice, the output is the same. One different letter gives two completely different fingerprints (f1 against f2), while the same content in two files with different names gives the same fingerprint (f1 against f3): the digest identifies the content, not the name. There is no "similar" version: either the content is identical, or the fingerprint is another. The registry uses the same principle on the image's manifest, which in turn lists the fingerprint of every layer: changing one byte in any layer changes the image's digest. In the OCI specification the digest is part of a content's "descriptor", next to the type and the size in bytes, and it is what allows you to refer to the content safely.

The trap: a tag can be moved

The miniature registry below is a teaching model, not Docker: it keeps two tables, the contents indexed by their digest (immutable) and the tags that point to a digest (movable). I publish version 0.1.0 in it and also call it latest; then I publish 0.1.1 and move latest onto it, as a careless release procedure would.

import hashlib, json

def digest(blob: bytes) -> str:
    return "sha256:" + hashlib.sha256(blob).hexdigest()

class MiniRegistry:
    """A miniature registry: contents are addressed by digest, tags are labels."""
    def __init__(self):
        self.blobs = {}   # digest -> content (immutable)
        self.tags = {}    # tag -> digest (movable)

    def push(self, tag: str, manifest: dict) -> str:
        blob = json.dumps(manifest, sort_keys=True).encode()
        d = digest(blob)
        self.blobs[d] = blob
        self.tags[tag] = d
        return d

reg = MiniRegistry()
v1 = {"layers": ["base", "dependencies", "app-v1"]}
v2 = {"layers": ["base", "dependencies", "app-v2"]}

d1 = reg.push("0.1.0", v1); reg.tags["latest"] = d1
print("after the first push   latest ->", reg.tags["latest"][:19])
d2 = reg.push("0.1.1", v2); reg.tags["latest"] = d2
print("after the second push  latest ->", reg.tags["latest"][:19])
print("0.1.0 ->", reg.tags["0.1.0"][:19], "| 0.1.1 ->", reg.tags["0.1.1"][:19])
print("by digest, the old version is still there:", json.loads(reg.blobs[d1])["layers"])

# after the first push   latest -> sha256:e8bea2880192
# after the second push  latest -> sha256:2b32ebd8b533
# 0.1.0 -> sha256:e8bea2880192 | 0.1.1 -> sha256:2b32ebd8b533
# by digest, the old version is still there: ['base', 'dependencies', 'app-v1']

The push method computes the digest of the content (the hash of the manifest serialized deterministically, with sort_keys=True) and records it both as content and as the target of the tag. The two lines with latest show the point: first it pointed to e8bea28..., then to 2b32ebd.... Same name, different content. The last line shows the other half: the old version has not vanished, it is reachable by digest. Run twice with Python 3.14.4, the output is identical. In Docker the behavior is the same: publishing a tag again reassigns it.

Whoever wrote docker pull churn-api:latest in a release procedure gets two different images depending on when they run it. For an ML service it is worse than for ordinary software: a different model can give different answers without anything flagging the error. Yesterday's deploy and today's, "with the same command", are not the same release; you cannot say with certainty which model is answering; and rollback becomes impossible, because "go back to latest" means nothing if latest has already changed.

Even a tag with a version number is not immutable by nature: it is so only if the registry is configured to forbid overwriting, or if the team respects it by discipline. The digest, instead, is immutable by construction.

Publishing and downloading, with a registry

The commands are these. They are a schema to adapt, not executed: the registry name, the user and the token are examples, and the outputs depend on the registry you use. The documentation of docker login provides the --password-stdin option, which takes the password or token from standard input.

docker login registry.example.com -u USER --password-stdin < token.txt
docker tag churn-api:0.1.0 registry.example.com/USER/churn-api:0.1.0     # gives a second name to the same image
docker push registry.example.com/USER/churn-api:0.1.0
# the last line of the output reports the digest computed by the registry:
#   0.1.0: digest: sha256:<fingerprint> size: <bytes of the manifest>      (deduced output, not executed)

The login reads the token from a file with --password-stdin, so that it appears neither in the shell history nor in the list of processes: a token on the command line is a secret that leaks (see environment variables and secrets). The tag builds the full name registry/owner/repository:tag and copies nothing: it is a second name for the same image, which has the same ID on your disk. This is why removing a tag with docker rmi name:tag does not delete the image as long as another name uses it. The push uploads only the layers the registry does not already have, and the final line is the one that counts: the digest, to be noted down. From the server, docker pull registry.example.com/USER/churn-api:0.1.0 is enough.

Pulling by digest

The syntax with @ asks the registry for exactly that content, without going through the tags:

docker pull registry.example.com/USER/churn-api@sha256:<fingerprint>
# deduced output, not executed: the image is downloaded even if `latest` points elsewhere

It is the property you need for a reproducible release and a certain rollback: the line you write in the release log identifies the content, and whoever runs it six months from now gets the same image. Note also the distinction between an image's local ID, computed on the configuration that sits on your disk, and the registry's digest, computed on the manifest: they are two different fingerprints. To pull "exactly that one" you use the registry's digest, which docker push prints.

A tag policy that works

The operating rule: in production you deploy a version tag or, better, a digest; you record which one you released; the latest tag serves at most for convenience in development. The same holds for the base images in Dockerfiles: in the article on the Dockerfile UV is taken from uv:0.12, not from latest, for the same reason. The strictest form is to give the digest in the FROM instruction too:

FROM python:3.12-slim@sha256:<published-digest>

This way the base does not change even if the 3.12-slim tag is updated with a security patch. It is the most reproducible choice, with a cost: security updates do not arrive on their own, you must change the digest when you decide to. A classic compromise between stability and freshness; often the digest is pinned and updated at regular intervals with an automatic check, for example a bot that proposes the change.

A tag convention that can be automated

Combining version and commit gives tags that are readable and traceable. A common usage: semantic version for releases (1.4.0) and short commit hash for every build (sha-a1b2c3d). With Git:

COMMIT=$(git rev-parse --short HEAD)
docker build -t churn-api:sha-$COMMIT .
# the first line saves the abbreviated hash of the current commit; the second (not executed) uses it in the tag

git rev-parse --short HEAD prints the abbreviated hash of the current commit. The result is an image whose tag says exactly which code it was born from; if sha-a1b2c3d is in production, you know which commit to read to understand what it does. ML case: also add the model version (sha-a1b2c3d-model-3), if the model is inside the image. An ML service image is defined by three things that change at different speeds: the code (often), the dependencies (rarely) and the model (at every retraining). A convention that makes them all readable:

What matters is that a person can answer "what is running in production?" by reading a tag, and that the same tag is never reassigned. Safer still, the release is recorded in the release log with the digest, not only with the tag; the model version is handled by the article on the model registry.

The real case: the commit as digest

This project has no registry because it has no images (the article on containers explains why): the release is a git pull on the server. Yet the problem is the same, and the solution has the same shape. The commit hash is the digest of the code: it identifies that exact content, and if one character changes, the hash changes. The rollback procedure in the deploy runbook (section 5.2 of docs/01_foundation/04_deploy/11_deploy_manuale.md) contains the line git checkout <COMMIT_HASH>, followed by the redeploy: it works because the hash does not move. The counterpart of latest would be to deploy "the branch as it stands right now" without noting which commit it is, and it is the same fragility. When you reconstruct what is running, you need an identifier that does not change: a commit hash, a digest, the version number of a model in the registry.

How a service gets from your computer to the server

You have an image that works on your laptop. The server is another machine. How does it get there? There are three roads, and the registry is the most structured. Seeing all three clarifies when the extra complication makes sense.

Road 1: build on the server. You copy the code to the server (with Git, see GitHub and pull requests) and run docker build there. Pro: no extra infrastructure. Con: the server must have Docker and the build tools, the build consumes its resources (with possible effects on the production service running next to it), and the image that runs is built there, so it is not the same one you tested on your computer: same Dockerfile, but a new moment, hence potentially different base versions. It is the analogue of this project's git pull plus build: a sensible choice with a single machine, but to be known as such.

Road 2: move the image as a file. Docker can export an image to a file and import it elsewhere, without a registry (docker save and docker load, documented in the CLI reference):

docker save churn-api:0.1.0 | gzip > img.tar.gz     # exports and compresses
ls -l img.tar.gz                                   # the size depends on the image's dependencies
docker rmi churn-api:0.1.0                         # (test) I remove the image
docker load < img.tar.gz                           # expected output: Loaded image: churn-api:0.1.0 (not executed)

After removing the image from the computer, docker load restores it identical. You transfer the file with scp or a disk, and import it on the server. Pro: the same image you tested, no registry. Con: you transfer everything every time (the whole file even for a one-line change), and you have no catalog, no shared tags and no history: everything is manual. For a single machine and rare releases it is acceptable.

Road 3: the registry. You publish once, the server (or servers) download what they need. Pro: a single authoritative place, tags and digests, incremental transfers, integration with pipelines. Con: one more service to run (or pay for), with credentials to manage. The choice follows the criterion already seen: how often and by how many people or machines you release. This project, with one machine and one person, has no images and therefore no registry either; a team with three services, two environments and a pipeline does.

Inside a registry: manifests and shared layers

Why is publishing the second version of an image fast? Because the registry does not store "whole images", but layers addressed by their hash, and a manifest that lists which layers make up that image. The same miniature model shows the effect: two versions that differ only in the code layer share the others.

# shared layers: each layer is identified by the fingerprint of its own content
def layers(app: bytes):
    return [digest(b"base system"), digest(b"python libraries"), digest(app)]
a = set(layers(b"app.py version 1"))
b = set(layers(b"app.py version 2"))
print("shared layers:", len(a & b), "of", len(b), "| new to upload:", len(b - a))
# shared layers: 2 of 3 | new to upload: 1

The listing uses the digest function from the previous block; a & b is the intersection of the sets of fingerprints. Two layers out of three coincide, and only the one with app.py is new: it is what a real registry does, uploading only the layers it does not already have. It holds on download too: a server that already has 0.1.0 downloads from 0.1.1 only the missing layer. This is the concrete reason why the order of the instructions in the Dockerfile matters even after the build: what rarely changes sits at the bottom and is not retransmitted. The real sizes depend on the image: you read them on your registry, or with docker history.

A consequence for machine learning: if the model is inside the image and weighs hundreds of MB, every retraining produces a new layer of that size to publish and download, even if the code has not changed. A good reason, in projects with large and frequently retrained models, to keep the model outside the image and let the image weigh only when the code changes.

Releasing and going back: comparing the three approaches

The row on going back is the one that weighs: with the git + build road, rollback requires rebuilding from the previous commit, so time and the risk of a build that no longer succeeds (a dependency no longer available). With a registry, rollback is a pull of something already built and already tested. It is the main advantage, and the one that pays for the complication when releases become frequent or critical.

When a registry is really needed

• It is needed if several machines or environments must have the same image, if the pipeline builds in one place and releases in another, or if you use a managed cloud, which starts from an image in its own registry.
• It is not needed if you build and use the image on the same machine, or if you use no images at all.
• Public or private: a public registry exposes your images to anyone, and with them everything you left inside. For your own code and models, private.

The three items answer three different questions: whether you need to share (the first), whether you need a central place (the second, by negation), and who can read (the third). The most underrated is the third: an image with a proprietary model, internal data or a secret copied by mistake (the article on secrets covers it) becomes public the moment you publish it on a public registry.

Security and access

• Who can write: if anyone can publish, anyone can change what runs in production. Limit writing to whoever (or whatever: a pipeline) has to release.
• Who can read: a private registry protects what is in the images.
• Limited-scope tokens: the pipeline uses a token that can only publish to that repository, short-lived and revocable, not the administrator's password.
• Scanning and signing: many registries offer vulnerability scanning on published images (see the article on open source licenses for the reasoning on dependencies), and tools exist to sign images and verify the signature before starting them; we do not go into them here.

The common thread is the principle of least privilege: every person and every automation has the strictly necessary permission. A pipeline token that can only publish to one repository, if it leaks, lets someone ruin that repository but not read everything else or administer the account.

Cleanup and cost

Images pile up: every build produces layers that take up space in the registry and on the server. On the computer or on the server, docker image prune removes unnamed (dangling) images, while with -a it also removes those not referenced by any container, and then you will have to download them again when needed (the command reference says so); docker system df shows how much space images, containers and cache take. In the registry you need a retention policy, for example "keep the last twenty images and all those with a version tag", because the space, often paid for, grows by itself with every continuous-integration build. Never delete an image if you do not know whether it is in production: keep a reliable release log (the article on deployment covers it) and consult it first.

Public base images (python:3.12-slim) are a dependency like any other: they have a maintainer, a license, known vulnerabilities. Three minimal habits: prefer official or verified images; pin the version and, in serious projects, the digest; rebuild periodically to pick up security fixes. Rebuilding is not the same as never changing: a CI pipeline that rebuilds every week and reruns the tests (the article on manual deploy and CI/CD covers it) is the way to update without releasing blindly.

Questions people really ask

"If I publish the same tag twice, what happens?" It depends on the registry: in general the tag is reassigned, as in the miniature registry. Many production registries have an option to forbid it (immutable tags): if it exists, enable it for version tags.

"Can I delete a tag?" Yes, but the image remains, identified by the digest, until no tag references it and the registry runs its cleanup. Whoever had pulled it by digest keeps working.

"How many versions should I keep?" At least: the one in production, the previous one (for rollback) and the latest within a reasonable window. The rest is cost.

"Do I have to use Docker Hub?" For your own projects, a registry integrated with the platform where you keep the code is often more convenient, because the permissions follow those of the repository. For public images (your bases), Docker Hub is the natural place. What matters is that the choice is deliberate and written down.

Try it yourself: four guided exercises

Exercise 1: moving latest

Save the miniature registry from this article in a file mini.py, run it and add a third push, 0.1.2, assigning latest to it again. You should see three different digests for the three version tags and latest equal to the last. With a real registry and an image of your own you can repeat the test with docker push and read the digest with docker buildx imagetools inspect or from the final line of the push. What you learn: that the tag moves and the digest does not.

Exercise 2: pulling by digest

With a registry at your disposal, remove the local tag (docker rmi name:0.1.0) and pull the old version again with its digest: docker pull registry/name@sha256:.... You should see the image come back, even if latest points elsewhere. What you learn: that a release by digest is reproducible and allows a certain rollback.

Exercise 3: a hash with your own hands

Create two text files that differ by a single character and compute sha256sum of both. Then repeat with the same content in two files with different names. You should see completely different fingerprints in the first case and identical ones in the second. What you learn: that the digest identifies the content, regardless of the name.

Exercise 4: shared layers

Modify the layer listing: add a fourth layer (for example b"model-v1") and compare two versions that change only the model. You should see three layers shared out of four. Then change the libraries layer and observe how many remain equal. What you learn: why the second publication is fast, and why the order of the Dockerfile instructions (what rarely changes first) matters.

What you can do now

You can read the full name of an image (registry, repository, tag or digest) and explain why the tag is a movable label while the digest is the fingerprint of the content, with a test you can repeat in a few seconds with sha256sum. You can choose a tag policy (version, commit, model), release by digest when reproducibility is needed, and write in the release log what is really running. You know when a registry is needed (several machines, several people, a managed cloud) and when it is not (one machine, one person, as in this project), and how to limit who writes and who reads. The principle holds beyond Docker: an immutable identifier, like a commit hash or the version of a model, makes rollback possible. The next step is tracking the experiments that produce the model: MLflow, tracking experiments instead of remembering them.