109 - ML-Ops: containers, images and isolation

Containers: understanding images, containers and isolation

A container is a process (or a group of processes) that runs on your computer but sees a world of its own: a filesystem of its own, a network of its own, a list of processes of its own. The filesystem comes out of an image, a read-only package that contains everything the program needs: the Python interpreter, the libraries, your code. The advantage concerns the fourth implicit dependency of the notebook, the operating system and the system packages: if you pack them together with the application, "it works on my machine" becomes "it works wherever there is a container engine". In this article you understand what they really are, without legends.

A preliminary note on how the commands were verified. The blocks that use Docker are declared in the text as deduced output, not executed: I wrote them on the basis of the official documentation and I explain their behavior, but I did not start them. The blocks that use only Linux tools (unshare, kill, the shell) I did run for real, twice, and they show the same kernel mechanisms that Docker relies on.

The words of containers

Six terms that appear on every page of documentation, and that are worth fixing before the commands:

• Kernel: the core of the operating system, the part that manages memory, processes and disks. Like the engine and the chassis of a bus: there can be different bodies on top, but the engine is one.
• Image: the read-only package that contains the base system, Python, libraries and code. Like the installation file of a game.
• Container: a running instance of an image. Like the game started from the installation file: you can start as many as you want from the same file.
• Layer: each build instruction of an image adds a layer of changes on top of the previous one, like the overlapping transparent sheets of an anatomical atlas.
• Volume: a folder that lives outside the container and survives it. Like the locker at the gym: you take the bag home, what you left in the locker stays.
• Namespace and cgroup: the two Linux kernel features that containers are based on. Namespaces decide what the process sees (its own files, its own processes); cgroups decide how much it can use (memory, processor).

The first four entries describe the objects you handle (image, container, layer), the last two the mechanisms that make them possible. If you remember just one pair, remember image and container: confusing them leads to the most common mistakes, such as trying to save a model inside a container and finding it gone. Dockerfile, daemon and ports appear later, at the point where they are needed.

The structure of an image and the container that is born from it:

The Docker overview defines the container as a "loosely isolated" environment: isolation exists but it is a property to configure, as we will see.

The analogy: the shipping container

Before standard containers, loading a ship meant handling crates of different shapes, each with its own rules. The metal container changed everything: inside you can put anything, but outside it always has the same shape, and cranes, trucks and ships treat it all the same way. The software container does the same: inside there is your application with its dependencies, outside it has a uniform interface (it starts, it stops, it exposes a port). The analogy has an important limit: the ship's container is a box, the software one is not a virtual machine. We see it right away.

The why: dependency hell and the idea of packing everything

Imagine having to run two projects on the same server: one requires one version of a system library, the other the next version, incompatible. With a single shared operating system, one of the two has to give way. Anyone who has administered servers knows this situation, called "dependency hell". Python virtual environments (uv and package management) solve it for Python libraries, but not for what lies below: the system libraries, the compiler, even the version of the interpreter.

Virtual machines solve everything, but at a price: each project gets an entire operating system, with its own kernel and its own reserved memory. The container is born as a middle way: isolating projects without duplicating the operating system. Each program sees a world of its own (its own files, its own processes, its own network), but underneath there is a single shared kernel. It is like an apartment block compared with separate houses: the walls divide the flats, but the foundations are common.

A container is not a virtual machine

A virtual machine simulates an entire computer, with its own operating system and its own kernel. A container shares the host's kernel and uses two features of the Linux kernel to give it the illusion of being alone: namespaces, which the namespaces(7) manual page describes as the abstraction that makes a group of processes appear to have their own isolated instance of a global resource, and cgroups, which the cgroups(7) page describes as the way of organizing processes into groups whose use of resources can be limited and measured. The first consequence is verified with a Docker command:

uname -r
docker run --rm python:3.12-slim sh -c 'uname -r; python -V; head -2 /etc/os-release'
# deduced output, not executed:
# the first line of the container is IDENTICAL to the line of `uname -r` on the host (same kernel);
# Python and the distribution are those of the image, not those of the host

The command starts a throwaway container (--rm) from the python:3.12-slim image and runs three commands in a row. The first, uname -r, prints the kernel version: inside the container it is the same as the host's, because there is no kernel of its own. The other two show the version of Python and the name of the distribution instead, which belong to the image. Practical consequences: a container starts in fractions of a second (no operating system to boot) and weighs much less than a virtual machine, but the isolation is weaker, because a kernel flaw is shared. A Linux image on a Linux host works natively; on Windows and macOS Docker Desktop quietly starts a small Linux virtual machine, which provides the kernel.

Isolation, seen with your own eyes

"A container sees a world of its own" becomes concrete with namespaces, which you can try without Docker with the unshare command of util-linux (unshare(1): runs a program in new namespaces). The block below was run on Linux with util-linux 2.41.3, twice:

echo "host: $(ls /proc | grep -cE '^[0-9]+$') processes"
unshare -r -p -f --mount-proc sh -c 'echo "namespace: $(ls /proc | grep -cE "^[0-9]+$") processes"'
unshare -r -u sh -c 'hostname test-ns; hostname'
hostname
id -u; unshare -r sh -c 'id -u'
# host: 502 processes        (498 on the second run: it changes continuously)
# namespace: 4 processes
# test-ns
# <host name>
# 1000
# 0

What it says, command by command. /proc is the special Linux folder in which every process has a numbered subfolder; counting them amounts to counting the visible processes. The host has roughly five hundred (the number changes from one machine and one run to the next), while the process started with -p (new process namespace) and --mount-proc sees four: the shell and the commands of its pipeline, and nothing more. With -u (new hostname namespace) the name changes only inside, and the host does not notice. With -r (new user namespace, mapping the current user to root) the identifier is 0 inside and 1000 outside: it is the idea by which, in a container, a process "is root" without being so on the host. A Docker container uses the same mechanisms, plus a filesystem and a network of its own.

The same comparison in Docker has this form:

docker run --rm python:3.12-slim sh -c 'echo "hostname: $(hostname)"; echo "processes: $(ls /proc | grep -cE "^[0-9]+$")"; ls /'
# deduced output, not executed:
# hostname: a hexadecimal identifier (the start of the container ID), different from the host's
# processes: a small number, few processes (as in the test with unshare)
# ls /: a filesystem of its own (bin, etc, usr, var...), that of the image

The host name inside a container is by default the shortened ID of the container, and the filesystem is that of the image: this is why a container does not see your files unless you bring them in with a volume.

Image and container: the recipe and the dish

The image is immutable: a stack of read-only layers, the recipe plus the ingredients already measured. The container is a running instance of an image, with a thin writable layer on top: from the same image you can start a hundred of them. The writable layer disappears when the container is removed, as the pair of commands that follows shows.

docker run --rm python:3.12-slim sh -c 'echo hello > /tmp/f.txt; ls /tmp'
docker run --rm python:3.12-slim ls /tmp
# deduced output, not executed:
# first command:   f.txt
# second command:  (no output: /tmp is empty, the file was in the writable layer of the first container)

The first container writes a file in /tmp and lists it; the second, started from the same image, starts from zero and does not find it. For an ML service this carries a golden rule: a container is throwaway. You do not save data, trained models or logs you want to keep in it. What must survive goes into a volume or into an external service.

The commands you need

The flags of docker run are documented in the command reference. The ones you will use most are -d and --rm for the life cycle, -p and -v to open passages to the outside, -e for configuration.

Ports and volumes: the two holes in the wall

By default the container is isolated: from the outside it cannot be reached and it does not see your files. Two options open a controlled passage. -p 18000:8000 connects port 18000 of your computer to 8000 of the container: the application inside listens on 8000 and knows nothing about 18000. -v "$PWD/vol:/data" makes the vol folder visible as /data in the container, and what is written there survives.

docker run --rm -v "$PWD/vol:/data" python:3.12-slim sh -c 'echo hello > /data/f.txt'
ls -la vol
# deduced output, not executed:
# -rw-r--r-- 1 root root ... f.txt     (owner: root)

Look at the expected owner: root. By default the process in the container runs as root (uid=0), and the files it creates in the volume belong to root on your computer too: afterwards you cannot modify them without sudo. It is a classic trap, and one of the reasons why in the article on the Dockerfile the service runs with an unprivileged user. With unshare -r you saw above a user who is 0 inside and 1000 outside, but that is a mapping of the user namespace; in the case of the volume the file is written on the host's disk with owner number 0.

A limit that surprises: the container is empty

docker run --rm python:3.12-slim python -c "import sklearn"
# deduced output, not executed:
# ModuleNotFoundError: No module named 'sklearn'

The base image has Python and almost nothing else: it is the "slim" variant, designed to be small. You add your libraries by building an image of your own, and that is the subject of the article on the Dockerfile. The message is the same ModuleNotFoundError you would see with an empty virtual environment: the container is a clean environment, which is both its merit and its cost.

Layers and cache: why order matters

An image is built by layers, one per instruction. Docker reuses a layer from the cache if the instruction and the files it uses have not changed; as soon as one changes, all the following layers are rebuilt. Practical rule: at the top of the Dockerfile what changes rarely (the dependencies), at the bottom what changes often (your code). If you copy the code before installing the dependencies, every one-line change invalidates the installation as well, and the rebuild repeats minutes of useless work.

The "layer" abstraction becomes concrete with docker history, which lists the layers of an image with the size of each:

docker history python:3.12-slim --format '{{.Size}}\t{{.CreatedBy}}'
# outline of the output, deduced output, not executed (sizes depend on the tag and the date):
# <tens of MB>   debian.sh --arch ... <- the base operating system
# <a few MB>     RUN ... apt-get install ... <- system packages needed by Python
# <tens of MB>   RUN ... (compilation and installation of Python)
# 0B              ENV PYTHON_VERSION=...   (variables: metadata only)
# 0B              CMD ["python3"]

It reads from the bottom up, from the oldest to the most recent. The instructions that weigh 0B are metadata (environment variables, the start command): they add no files. The size of an image is the sum of its layers, and two images that share the base layers also share them on disk: this is why ten images built on python:3.12-slim do not weigh ten times its weight.

The real case: where a container is needed and where it is not

An honest exercise is to look at this repository. The three services (Spring Boot, Angular SSR, FastAPI) do not have a Dockerfile: git ls-files finds Dockerfiles only among the digital products on sale, not in the site's services. The three services run as processes under systemd on a VPS, with PostgreSQL and Redis installed natively. Containers appear elsewhere: two docker-compose files in infra/ for auxiliary tools (Lighthouse CI and monitoring) and the backend's integration tests, which start real dependencies with Testcontainers (backend/pom.xml declares the library) and therefore require Docker.

It is not an oversight, it is a calculation. The container solves "same environment on many machines, released by different people". Here there is one machine, one person who releases, and an environment built once. The cost of the container (images to build and keep, networks, base-image updates, one more level in debugging) exceeds the benefit. Where instead you need an ephemeral and identical service, such as a database for a test, the container is the right tool.

The table is a criterion, not a rule: the right-hand column changes with the number of machines and of people involved, not with the fashion of the moment. If your situation is the first row, the container is a complication you can postpone; if it is the second or the third, it saves you more than it costs.

A real container: starting it, observing it, getting inside

docker run -d --rm --name web -p 18002:8000 python:3.12-slim python -m http.server 8000

What it says, word by word: -d in the background; --rm removes it when it ends; --name web gives it a name you can use in the following commands; -p 18002:8000 connects port 18002 of your computer to 8000 of the container; then the image and finally the command to run inside (a minimal web server included in Python, listening on 8000).

curl -s -o /dev/null -w "http=%{http_code}\n" localhost:18002/
docker ps --format '{{.Names}}\t{{.Image}}\t{{.Ports}}'
docker logs web | tail -2
# deduced output, not executed:
# http=200
# web   python:3.12-slim   0.0.0.0:18002->8000/tcp
# a server log line for the request just made, "GET / HTTP/1.1" 200

The curl on the port of your computer receives 200 because the traffic reaches the server inside the container. docker ps shows the name, the image and the port mapping. docker logs reports what the program wrote to standard output: this is why a containerized service must write its logs to standard output and not to an internal file, because docker logs and the platforms collect them from there. To stop it, docker stop web: thanks to --rm the container disappears, and docker ps -a no longer lists it.

docker exec runs a command in an already started container, like opening a window onto its inside (with -it and sh you get an interactive shell). Minimal images have less attack surface and less weight, but also fewer tools for debugging: in a "slim" variant even a command such as ps may be missing, and in that case you inspect from the host or use a diagnostic image. The trade-off is real.

The limits: what happens when the container overdoes it

A container without limits can consume all of the host's memory: the documentation on resource limits says that by default a container has no constraints and can use as much as the kernel scheduler grants it. Cgroups make it possible to set a ceiling. The following command tries to allocate 200 MB in a container limited to 64 MB:

docker run --rm --memory=64m --memory-swap=64m python:3.12-slim \
  python -c "x = bytearray(200*1024*1024); print('allocated')"
echo "exit=$?"
# deduced output, not executed: no "allocated" on standard output and exit=137

--memory=64m sets the ceiling; an equal --memory-swap prevents exceeding it by using the disk (the docker run reference defines it as memory plus swap). The program is terminated by the system and the exit code is 137, which means 128 + 9: the shell, when a command ends because of a signal N, returns 128+N (the bash manual page says so), and signal 9 is SIGKILL. The 137 does not prove exhausted memory by itself, it is the sign of a kill with SIGKILL; the typical cause in a container is precisely the memory ceiling being exceeded.

The code 137 can be reproduced without Docker, and it is the only piece of this block that I ran (twice):

sh -c 'kill -9 $$'; echo "exit=$?"
# exit=137

The child shell kills itself with signal 9 ($$ is its own process number), and the parent shell reads 128+9. For an ML service the situation is concrete: a large model, loaded several times (one more worker) or a data batch that is too wide can exceed the limit, and the service "vanishes" without a Python error message. If you see a container that ended with code 137, check the memory before the code.

You can tighten further: read-only filesystem, no network

Isolation is a configuration, not all or nothing. Two experiments show how far it can be tightened, with the --read-only (read-only root filesystem) and --tmpfs (a folder in memory) options of the docker run reference:

docker run --rm --read-only python:3.12-slim sh -c 'touch /x'
docker run --rm --read-only --tmpfs /tmp python:3.12-slim sh -c 'touch /tmp/x && echo "tmp writable"'
# deduced output, not executed:
# touch: cannot touch '/x': Read-only file system
# tmp writable

--read-only makes the container's filesystem non-writable: if an attacker managed to run commands in the service, they could not install anything or modify the program's files. But an application often needs a temporary folder, and --tmpfs /tmp mounts a small area in memory just for that. With the network the option is --network none:

docker run --rm --network none python:3.12-slim python -c "import urllib.request as u; u.urlopen('http://example.com', timeout=3)"
# deduced output, not executed: urllib.error.URLError, with a name resolution error
# (the exact text depends on the Python version and the system library)

--network none removes all network access: the program cannot even resolve names. For a batch job that reads local files and writes local files it is an excellent defense: even if compromised, it has no way of sending data elsewhere. They are three levers (non-root user, read-only filesystem, limited network) that progressively tighten the perimeter. It is the same logic as the NoNewPrivileges, ProtectSystem and PrivateTmp options of this site's systemd unit portfolio-ml.service, obtained in another way (the article on deployment discusses it).

The network trap: "localhost" in the container is not your localhost

One of the most frequent beginner errors. The service in the container has to talk to another service on your computer (a database, MLflow, Redis). It writes localhost:5432, and it does not work. The reason: every container has its own network, and inside it localhost means the container itself, where nothing is listening on that port.

# on the computer: a service listening only on 127.0.0.1:18077
docker run --rm python:3.12-slim python -c "import urllib.request as u; u.urlopen('http://localhost:18077')"
docker run --rm --network host python:3.12-slim python -c "import urllib.request as u; u.urlopen('http://localhost:18077')"
# deduced output, not executed:
# first command:   URLError (connection refused: inside the container there is nothing on 18077)
# second command:  reaches the service (the container shares the host's network)

With --network host the container shares the host's network and "localhost" goes back to meaning your computer. It is not the solution to prefer in general, because it removes network isolation, but it clarifies the mechanism. The correct routes depend on the case. If the two services are both in containers, make them talk by service name on the same Docker network (with Compose it is automatic: the db service is reached at the address db). If the service on the host has to be reached from the container, use an address the container can resolve: the docker run documentation reports the convention host.docker.internal, which Docker Desktop resolves on its own and which on Linux has to be associated with --add-host=host.docker.internal:host-gateway; the service on the host must listen on a reachable interface, not only on 127.0.0.1. In any case the address goes in the configuration (see environment variables), not in the code, so that it changes between development and production without modifications.

Several containers together: Docker Compose

A real service often has several parts: the prediction service, a database, MLflow, a monitoring system. Starting and connecting several containers by hand is tedious. Docker Compose lets you describe them all in a YAML file and start them together with one command. This site's repository has two in infra/, and it is instructive to read infra/monitoring/docker-compose.monitoring.yml (189 lines, counted with wc -l). The header says (in Italian): "Avvia Prometheus + Grafana + Loki + Node Exporter + Promtail su Docker. Le app (backend, ml, frontend) girano nativamente come systemd — solo il monitoring usa Docker.", that is: it starts Prometheus + Grafana + Loki + Node Exporter + Promtail on Docker; the apps (backend, ml, frontend) run natively under systemd, and only monitoring uses Docker. The command documented in the same file is docker compose -f infra/monitoring/docker-compose.monitoring.yml up -d. The elements the file declares are those of this article, translated into YAML:

• services: the list of containers: node-exporter, prometheus, alertmanager, loki, promtail and grafana, each with its image and its options;
• networks: a dedicated network of the bridge type, called portfolio-monitoring, where the containers find each other by name;
• volumes: three local volumes (prometheus_data, grafana_data, loki_data) so that the monitoring data survives the container;
• the published ports, in this file bound to 127.0.0.1 (9090, 9093, 3100 and 3000), hence reachable only from the server itself: access from outside goes through the Nginx proxy, as the header says.

This is a perfect example of the rule "the container is for where it is needed": a set of third-party tools (Prometheus, Grafana, Loki), with complex installations and dependencies of their own, is much simpler to start as ready-made containers than to install one by one on the server. The project's three services, written by the person who manages them, with a single environment to maintain, stay under systemd. The choice is not inconsistent: it is the same criterion applied to two different situations. Note also the detail about ports: binding the port to 127.0.0.1:9090:9090 instead of 9090:9090 is an isolation choice, because in the second case the port would be reachable from any interface of the host.

Containers for machine learning: what changes

Four particulars to keep in mind when the content is an ML service. Images are large: ML libraries bring heavy binaries with them, and with deep learning libraries and GPU support you reach many gigabytes, with an effect on build times, publishing and cold start (the size depends on the libraries and must be measured on the image you build). Models are large too: including them in the image or mounting them as a volume is a choice with compromises between reproducibility and speed of updating. GPUs require explicit configuration: the container does not see the graphics card unless you use the manufacturer's runtime and declare it, and the image must contain libraries compatible with the host's drivers, which are part of the shared kernel. Finally, startup is slow: loading a model takes seconds or minutes, and a container declared "unhealthy" before it has finished loading the model is restarted in a loop by the orchestrator.

The last point has a counterpart in this repository: the systemd unit of the ML service has TimeoutStartSec=300 (infra/systemd/portfolio-ml.service), that is it grants up to five minutes for startup before considering it failed. The same reasoning applies to a container's health checks.

The shared kernel: practical consequences

Three limits descend from the shared kernel. An image built for the wrong architecture (for example ARM on an x86 host) does not start, or starts slowly emulated. The container cannot have a different kernel, so functions that require kernel modules or drivers (GPUs) need explicit configurations. And a kernel flaw affects all the containers on the host: this is why isolation between different customers is done with virtual machines, not with containers alone. The common thread is that the container isolates processes, not the hardware nor the kernel beneath them.

Building locally and releasing elsewhere: the promise and its limit

The promise is "I build once, it works the same everywhere". It holds as long as host and image are compatible (same processor architecture, same kind of kernel) and as long as the image truly contains everything. What stays outside and can break the promise: the configuration (the environment variables passed at startup), the data (volumes) and the network dependencies (a database, an external service). The container packs the program, not its world.

How to reason when something does not work in a container

1. Does the container start? docker ps -a also shows the ones that ended, with their exit status. An exit with code 137 indicates a kill with SIGKILL, often for memory (seen above); 1 is usually an application error.
2. What does the log say? docker logs NAME, even after it has ended (as long as you have not removed it).
3. What does it see inside? docker exec -it NAME sh to inspect files and variables; docker run --rm -it --entrypoint sh IMAGE to enter an image without starting the service.
4. Does the configuration arrive? docker inspect NAME shows variables, mounts, network.
5. Is the problem the network? Remember the localhost trap.
6. Is the problem the build? docker build --progress=plain . shows the whole log, and --no-cache repeats every step from scratch if you suspect a stale cache.

The order goes from the cheapest to the most expensive: the status and the log are read in a few seconds and solve most cases, internal inspection requires a live container, rebuilding without cache costs minutes. Start from the first question and go down only if the answer is not enough.

Try it yourself: four guided exercises

Exercise 1: counting what the process sees

Run ls /proc | grep -cE '^[0-9]+$' on your computer, then the same command inside unshare -r -p -f --mount-proc sh -c '...' and, if you have Docker, inside docker run --rm python:3.12-slim sh -c '...'. You should see hundreds of processes on the host and very few in the namespace or in the container, with a different hostname in the container. What you learn: process isolation, concretely.

Exercise 2: limits and read-only filesystem

With Docker, run the program that allocates 200 MB with --memory=64m --memory-swap=64m: you should see an exit with code 137 (echo $?). Then docker run --rm --read-only python:3.12-slim sh -c 'touch /x': Read-only file system; add --tmpfs /tmp and check that /tmp is writable. What you learn: that 137 is the sign of a kill with SIGKILL and that isolation is tightened with a few options.

Exercise 3: the localhost trap

Start on the computer python3 -m http.server 18077 --bind 127.0.0.1. From a container try to reach http://localhost:18077 with urllib: you should see a connection error. Repeat with --network host: this time it answers. Then explain, in your own words, why in the first case "localhost" was not your computer. What you learn: the cause of the most common beginner network error.

Exercise 4: opening an image

Run docker history python:3.12-slim and docker image inspect python:3.12-slim --format '{{len .RootFS.Layers}}'. You should see few layers and many 0 B instructions: count how many and which weighs the most. Then build a small image of your own with two RUN instructions and observe how they appear in the history. What you learn: that an image is the sum of its layers and that metadata-only instructions weigh nothing.

Recommended cleanup at the end of the exercises: docker ps -a to see if anything is left, docker image prune for unnamed images, and docker system df to know how much space they take up.

What you can do now

You can say why a container is not a virtual machine: it shares the host's kernel and uses namespaces and cgroups to isolate what the process sees and how much it consumes. You can tell an image from a container, and you know that the writable layer disappears with the container, so that what must remain goes into a volume. You can order a Dockerfile to exploit the cache, recognize an exit 137, avoid the localhost trap and tighten isolation with a non-root user, a read-only filesystem and a limited network. And you can judge whether the container is useful to you today: where there is one machine and one person, often it is not. The next step is to build an image of your own: a Dockerfile for an ML service.

The same series also includes ML-Ops: from notebook to service and ML-Ops: container registries, tags, digests and versions that do not change under your feet.