108 - ML-Ops: environment variables and secrets outside the code

Environment variables and secrets: configuration kept out of the code

An environment variable is a name-value pair (API_TOKEN=abc123) that the operating system makes available to every program it starts. The code reads it, and so exactly the same code behaves differently on your laptop, in the tests and on the server, without you modifying it. It is the third of the five implicit dependencies of the notebook described in the introduction to the module: the configuration (paths, addresses, keys). Here you learn to keep it out of the code, to validate it at startup and not to let it leak.

The words of configuration

There are few of them, but they are used loosely, and that causes real mistakes. The five you need in order to read the rest:

• Configuration: everything that changes between one environment and another without changing the logic (addresses, paths, limits, switches). Like the settings of a washing machine: the same appliance, different programs.
• Secret: a piece of configuration that gives access to something and must not be seen by anyone unauthorized (passwords, tokens, keys). Like the key to your house.
• Environment variable: a NAME=value pair that a process, that is a running program, passes to its children. Like a note a parent hands to a child before sending them out: the child reads it, but what the child writes on their own note never goes back to the parent.
• Validation: checking at startup that the configuration is complete and of the right type (a number where a number is needed), and stopping with a clear message if it is not.
• Rotation of a secret: replacing it with a new one and revoking the old one. It is the standard answer to a leak.

The distinction that holds up everything else is the first pair: configuration can sit in plain text in a repository file, a secret cannot. Almost all incidents on this topic start from a secret treated like any ordinary piece of configuration, that is written in a file that then ends up in Git, in a log or in an image. The other three terms describe the mechanism by which a program receives the values and the way it is defended against mistakes.

Where a program looks for the value of a setting, in order:

The scheme is the one used by pydantic-settings, the library we will use below: its documentation lists, in decreasing order of priority, the arguments passed to the class constructor, the environment variables, the dotenv file, the secrets directory and finally the default values. The constructor is missing from the drawing for simplicity: we find it again in the section on tests.

The analogy: the nameplate on the door and the key

The nameplate on the intercom ("Rossi, flat 4") can be written on the wall, everyone reads it. The house key cannot: you do not engrave it on the front door, you keep it in your pocket, and if you lose it you change the lock without rebuilding the front door. In software, the address of the database and the path of the model are the nameplate: configuration. The database password is the key: a secret. Both live outside the code, but the secret has stricter rules.

The why: three ways configuration inside the code does damage

"I'll put the database address directly in the code, for now" is one of the most expensive sentences in development. Configuration inside the code looks harmless because it works, and it does damage in three precise ways.

First: the wrong environment. You have a development address and a production one. If they are in the code, moving the program from one environment to another means modifying it, and whoever modifies sooner or later forgets to put things back. The classic result is a test program that writes to the production database, or a production one that reads fake data. External configuration makes the identical code able to run anywhere: what changes is the environment, not the program.

Second: the secret in the history. A password written in the code ends up in a commit; the commit ends up in the Git history (see Git: version your work); the history ends up on a remote (GitHub and pull requests) and in the copies of whoever clones. Even if you remove it afterwards, it stays in the old commits. The only truly effective action after a leak is to replace the secret. Keeping it out of the code from the first line avoids the whole problem.

Third: the change that requires a release. If the limit of requests per minute or the address of a service is in the code, changing it means a new version, tests, build, deploy. If it is in a configuration file, you change one line and restart. For the settings you adjust often (thresholds, limits, feature switches) the difference is enormous.

These three points are the substance of the third factor of the Twelve-Factor App, a set of principles for building services that run well in the cloud: "Store config in the environment". The test the document proposes to know whether the separation is done well is this: could the codebase become open source at any moment, without compromising any credentials?

The mechanism: every process has its own environment

When a program starts another, the child receives a copy of the parent's variables. The copy goes down, it does not come back up. The block below tests it in bash with a variable named DEMO, then shows the three forms in which a variable can be defined in a shell: in front of a single command, with export, and removed with unset:

(export DEMO=hello; sh -c 'echo "child sees: $DEMO"')   # child sees: hello
echo "parent sees: [$DEMO]"                              # parent sees: []
sh -c 'echo "1) [$ZZ]"'                       # 1) []         the variable does not exist
ZZ=hello sh -c 'echo "2) [$ZZ]"'             # 2) [hello]    defined only for this command
sh -c 'echo "3) after: [$ZZ]"'                # 3) after: []  gone: it only applied to that command
export ZZ=export; sh -c 'echo "4) [$ZZ]"'     # 4) [export]   exported: every child inherits it
unset ZZ; sh -c 'echo "5) [$ZZ]"'             # 5) []         removed
env -i /usr/bin/env | wc -l                   # 0

Run twice with bash, the block gave the same output reported in the comments. The first line shows that the child sees the variable and the parent does not. Lines 2 and 4 are the two forms to tell apart: NAME=value command applies only to that command, and is the safest form for a temporary secret because it does not stay in the session; export NAME=value makes it available to all the programs started from that shell, until you use unset or close the terminal. Without export, a variable defined in the shell does not pass to the children. The last line launches env with a completely empty environment (env -i) and counts the lines it prints: zero. It serves to discover that a program worked only because on your machine HOME or PATH happened to be there. To see what is defined, printenv NAME shows one variable, env alone lists them all.

From Python it is read with os.environ:

import os
token = os.environ["API_TOKEN"]              # KeyError if missing: it fails right away
porta = int(os.environ.get("PORT", "8000"))  # with a fallback value, ONLY if it makes sense

The difference between the two lines is a design decision. os.environ["API_TOKEN"] raises KeyError if the variable is not there; .get("PORT", "8000") returns the fallback value, and at that point int() converts the string to a number (environment variables are always strings). For a value without which the service cannot work, such as a token, its absence must be an error at startup. A fallback value is fine for settings that are truly optional, such as the port.

.env files: format and traps

Declaring ten variables by hand every time you open the terminal is inconvenient, so the convention of the .env file was born: NAME=value lines, blank lines and comments with #, which a library loads when the program starts.

# .env  (does NOT go in Git: it is in the .gitignore)
API_TOKEN=abc123
MLFLOW_TRACKING_URI=sqlite:///mlflow.db

Next to it you put a .env.example with the names but without the secret values, which does go in Git: it documents which variables are needed. Whoever clones the project copies the example, fills in the values and starts.

# .env.example
API_TOKEN=change-me
MLFLOW_TRACKING_URI=sqlite:///mlflow.db
MODEL_PATH=model.joblib

The format has a few subtleties that cause silent errors. With pydantic-settings 2.14.1 I read a file that has quotation marks around a value with spaces and a # inside, and spaces around an =:

# f.env
API_TOKEN="with spaces # not a comment"
MODEL_PATH = /tmp/m.joblib
# comment

# read with BaseSettings(env_file="f.env"):
# 'with spaces # not a comment'
# '/tmp/m.joblib'
#
# the same file read by the shell (. ./f.env):
# sh: 2: ./f.env: MODEL_PATH: not found

The token between quotation marks keeps its spaces and the # symbol, which outside the quotation marks starts a comment. The spaces around the = in the second line are tolerated by pydantic-settings, while the shell tries to run MODEL_PATH as a command and fails. The comparison says that different readers have different rules, and that containers and systemd have their own about quotation marks. Practical rule for a file read by several tools: no spaces around the =, values with spaces in double quotation marks, a single format for the whole project.

Validating at startup with pydantic-settings

Reading os.environ by hand, scattered through the code, leads to mistakes discovered late (a "8000" string where a number is needed). pydantic-settings concentrates the configuration in a typed class and validates it when the program starts:

# settings.py
from functools import lru_cache
from pydantic import SecretStr
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", extra="ignore")

    mlflow_tracking_uri: str = "sqlite:///mlflow.db"
    model_path: str = "model.joblib"
    api_token: SecretStr          # no default: if it is missing, startup fails

@lru_cache
def get_settings() -> Settings:
    return Settings()

Three lines matter. env_file=".env" says which file to read, and it is a path relative to the folder from which you launch the program. extra="ignore" avoids the error for variables present in the file but not declared in the class (without it, the default would be "forbid" on the dotenv file and the program would not start). api_token: SecretStr has no default: it is the required field, and SecretStr masks the value when the object is printed. lru_cache on get_settings() makes the configuration be read only once.

With this settings.py I ran a short script (prova.py: it prints the object and then the value of the token with .get_secret_value()) in three situations, twice each with the same outcome:

# 1) no API_TOKEN and no .env
# pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
# api_token
#   Field required [type=missing, input_value={}, input_type=dict]
#
# 2) .env with API_TOKEN=abc123
# mlflow_tracking_uri='sqlite:///mlflow.db' model_path='model.joblib' api_token=SecretStr('**********')
# abc123
#
# 3) the same .env, but API_TOKEN=from-the-shell set in the terminal
# mlflow_tracking_uri='sqlite:///mlflow.db' model_path='model.joblib' api_token=SecretStr('**********')
# from-the-shell

In the first case the service does not start and the message says which variable is missing. In the second it starts, and printing the object does not reveal the secret: to read it you must explicitly call .get_secret_value(), which the second line prints. In the third the shell wins over the file, as the priority order promises. This precedence has a practical reason: a value set in production is not overwritten by a forgotten file on disk. The block runs with Python 3.14.4, pydantic 2.13.4 and pydantic-settings 2.14.1.

Trap: a service that starts with a wrong configuration and gets things wrong silently is worse than one that does not start. An empty database address that falls back on a development default can make real data be written in the wrong place. Prefer the noisy failure at startup.

Types and validation: what pydantic-settings really does

Environment variables are always strings. The program wants numbers, booleans, lists. Manual conversion is easy to forget and to get wrong. pydantic-settings converts based on the declared types and rejects whatever does not convert. The CHURN_ prefix avoids collisions with the variables of other programs:

from pydantic import SecretStr
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="CHURN_", env_file=None, extra="ignore")
    port: int = 8000
    debug: bool = False
    allowed_origins: list[str] = []
    api_token: SecretStr = SecretStr("")

s = Settings()
print(s.port, type(s.port).__name__, s.debug, s.allowed_origins)
print(s.model_dump())
print(Settings(port=1, debug=True).port)

# CHURN_PORT=9000 CHURN_DEBUG=yes CHURN_ALLOWED_ORIGINS='["https://a.it","https://b.it"]' CHURN_API_TOKEN=abc
# 9000 int True ['https://a.it', 'https://b.it']
# {'port': 9000, 'debug': True, 'allowed_origins': ['https://a.it', 'https://b.it'], 'api_token': SecretStr('**********')}
# 1

The output in the comments is the one with the four CHURN_* variables set. CHURN_PORT=9000 becomes a real int, yes becomes True, and the list arrives from a string in JSON format, which is how pydantic-settings reads complex types. The last line shows that the constructor's arguments take precedence over the environment: Settings(port=1, debug=True) gives port=1. The model_dump() shows the masked token, so logging the whole configuration at startup (useful for debugging) does not write the secret into the logs. A good practice is exactly this: print the effective configuration at startup, with the secrets masked, to know with which values the service is running.

I then tried the cases that fail. With CHURN_PORT=eight startup stops with Input should be a valid integer, unable to parse string as an integer, indicating the field and the value. With CHURN_DEBUG=maybe the error is Input should be a valid boolean, unable to interpret input. For booleans, true, 1, yes, on give True, while 0 and off give False: useful, but the shell or another language do not use the same rules, so agree on one format. The prefix really isolates: with PORT=1234 without the prefix the program keeps using 8000, and a generic variable set by something else does not modify your service by mistake. An error at startup costs a minute; an "eight" that blows up in the middle of a request costs an incident.

Testing the configuration without touching the real environment

In tests you do not want to depend on the machine's environment. With monkeypatch (pytest) you set and remove variables only for the duration of the test:

import pytest
from pydantic import ValidationError
from settings import Settings, get_settings

@pytest.fixture(autouse=True)
def isolato(monkeypatch, tmp_path):
    monkeypatch.chdir(tmp_path)               # no real .env within reach
    monkeypatch.delenv("API_TOKEN", raising=False)
    get_settings.cache_clear()                # empties the lru_cache cache
    yield
    get_settings.cache_clear()

def test_manca_il_token_l_avvio_fallisce():
    with pytest.raises(ValidationError, match="api_token"):
        Settings()

def test_il_token_arriva_dall_ambiente(monkeypatch):
    monkeypatch.setenv("API_TOKEN", "test-secret")
    s = Settings()
    assert s.api_token.get_secret_value() == "test-secret"
    assert "test-secret" not in repr(s)    # it does not end up in the logs

# python3 -m pytest -q test_settings.py
# ..
# 2 passed

The fixture with autouse=True runs before every test: it moves the current directory into a temporary folder, so the project's real .env is not read, removes API_TOKEN from the environment and empties the cache. The cache_clear() is the typical trap: with lru_cache the configuration is read only once, and a later test would see that of the previous one. The second test also checks that the value does not appear in repr(), that is in what would end up in a log. The execution time (0.1 s on my machine) changes from one machine to another.

As a lightweight alternative, pydantic lets you pass the values directly to the constructor, and they take precedence over the environment (Settings(port=1, debug=True) in the previous block). The two techniques are used together: the constructor to test the logic that uses the configuration, monkeypatch to test the reading.

One more test keeps the .env.example honest, because an example nobody updates grows old. It compares the keys of the example with the fields of the class:

from settings import Settings

def test_env_example_elenca_tutte_le_variabili():
    chiavi = {r.split("=")[0].lower() for r in open(".env.example") if "=" in r}
    campi = set(Settings.model_fields)
    assert chiavi == campi

# with the complete example (three keys):  1 passed
# removing MLFLOW_TRACKING_URI from the example:
# E   AssertionError: assert {'api_token', 'model_path'} == {'api_token',... 'model_path'}
# 1 failed

The test reads the keys from the file (the part before the =, in lowercase), takes the names of the fields declared in the class (model_fields) and checks that they coincide. I ran it on the test folder: with the example from the article it passes, and removing a line from the example makes it go red, as it should. It must be launched from the project folder, where the .env.example is, because the path is relative. If you add a variable to the code and forget the example, the next run warns you.

The real case: how configuration reaches production

The ML service of this site shows two real practices, both in the repository. In the systemd unit portfolio-ml.service the line EnvironmentFile=/opt/portfolio/ml-service/.env.prod makes systemd read the file and pass the variables to the process, so the code does not know where they live; the systemd.exec documentation describes EnvironmentFile= and Environment=. The non-secret variables, such as the model cache folders, are declared in the same unit with Environment=. The module app/config.py then supports the NAME_FILE pattern: if a variable such as JWT_SECRET_FILE exists, the value is read from the content of that file, and otherwise from the variable JWT_SECRET. It is the convention of Docker and Kubernetes secrets, which mount the secret as a file rather than as a variable.

The essential parts of the configuration class, abbreviated:

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=f".env.{os.environ.get('APP_ENV', 'dev')}",
        env_file_encoding="utf-8",
        case_sensitive=False,
        extra="ignore",
    )
    redis_url: str = "redis://localhost:6379"

Line by line. env_file=f".env.{APP_ENV}": the file to read depends on a variable, APP_ENV, which by default is dev. So APP_ENV=prod chooses .env.prod, and the same code configures itself for each environment without changes. case_sensitive=False: REDIS_URL and redis_url are the same thing, a useful indulgence because environment variables are conventionally written in uppercase. extra="ignore": variables not declared in the class are ignored instead of raising an error, because variables of other programs can coexist in the same file. redis_url has a sensible default for development (localhost), while in production the file overrides it. This block is an excerpt, not a program that runs on its own.

The _read_secret function that implements the NAME_FILE pattern reads the file with .strip(), which removes trailing spaces and newlines: a detail that solves the error described below.

The secret with the newline: a real trap

When a secret is in a file, the file usually ends with a newline character, because most editors and commands (echo, printf '...\n') add it. The listing creates a file like that and reads it with and without .strip():

# printf 'tok123\n' > segreto
raw = open("segreto").read()
print(repr(raw))           # 'tok123\n'
print(repr(raw.strip()))   # 'tok123'

Without .strip() the secret read is tok123\n, not tok123: the comparison with the expected value fails, or a call to an external service answers "unauthorized" for a reason invisible to the naked eye, because a trailing space or newline does not show in the logs. It is one of the most frustrating configuration bugs. Rule: when an authentication "does not work" with a secret that looks right, check for trailing spaces and newlines, by printing repr(secret) (locally, on screen, for an instant: not in shared logs).

Where a secret leaks even when you use variables

"Environment variable" keeps the secret out of the code and out of Git, but does not make it invisible. The secret can come out from four points: from the logs and error messages, if you print the whole configuration (SecretStr helps, but it is not a universal guarantee); from the container image, if you write it with ENV in the Dockerfile; from the inspection of the process, because variables are readable by whoever administers the machine; and from child processes, which inherit the whole environment even if they should not see it. Three of these points can be seen with three commands.

1. Inside the container. By passing the variables with --env-file, anyone with access to the Docker engine reads them with docker inspect. It is not a defect: the variables are visible to the process, and therefore to whoever administers the process.

docker run -d --rm --name envt --env-file e.env python:3.12-slim sleep 20
docker inspect envt --format '{{range .Config.Env}}{{println .}}{{end}}' | grep API_TOKEN
# API_TOKEN=abc123   (output deduced, not executed)

This block is deduced output, not executed: the command requires Docker, absent from the test environment, so the output is the one the command produces according to its definition (.Config.Env is the list of the container's variables). Try running it yourself, with an e.env that contains API_TOKEN=abc123.

2. In the image, if you bake it in at build time. A Dockerfile with ENV API_TOKEN=abc123 leaves the value in plain text among the image's instructions, and docker history shows them. The build secrets documentation says it in one sentence: build arguments and environment variables are inappropriate for passing secrets to the build, because they persist in the final image; for the build you use secret mounts.

docker history envleak --no-trunc --format '{{.CreatedBy}}'
# ENV API_TOKEN=abc123   (line expected among those printed; output deduced, not executed)

This block too is deduced output, not executed. Anyone who can download the image from a registry (the article on registries discusses them) can read that line without starting anything. This is why secrets are passed at startup (docker run -e, --env-file), never in the Dockerfile. How to write a correct Dockerfile is the subject of the article on the Dockerfile.

3. In the operating system. On Linux every process exposes its own environment in /proc/<pid>/environ. This block I did run, twice:

MIO=secret123 sleep 5 &
tr '\0' '\n' < /proc/$!/environ | grep MIO
# MIO=secret123

The & sends sleep to the background, $! is its process number, and tr replaces the null characters that separate the entries with newlines. The proc_pid_environ manual page describes the file as the initial environment with which the program was started. The file is readable by the user who owns the process and by root: another reason why a service runs with a dedicated user (the article on deployment discusses it). Variables are not a safe, but a way to keep secrets out of the code and out of the Git history.

Which level of protection to choose

"Environment variable" is the base level. Going up in protection, and in complexity:

Which level? It depends on the cost of a leak and on who else has access to the machine. For a project with one person and one server, levels 1-2 are proportionate; for a team with several services on the same machine, or sensitive data, you go up. A useful criterion: the right level is the simplest one that makes rotating a secret a five-minute operation. If rotating a key requires a release or the modification of ten files, nobody will do it even after a leak.

A note of transparency on this site's project: its .env.prod files are tracked in Git because the repository is private, a deliberate choice, accepted for deploy simplicity. Compared with the table it is a "reduced" level 1 that holds as long as access to the repository is controlled; the same choice in a public repository would be an incident, because there a committed secret is a published secret. The levels are compromises that depend on the context, not moral ladders.

A working order for secrets

1. List what is secret (tokens, passwords, keys) and what is only configuration (addresses, paths).
2. Out of the code and out of Git: .env file in .gitignore, with git check-ignore -v .env to verify it.
3. Tight permissions on the server file: chmod 600 .env.prod (only the owner reads and writes).
4. One secret per environment: a development key that leaks must not open production.
5. Never in the logs (SecretStr helps) and never in the build.
6. A rotation plan: know where the secret is changed and who does it, before it is needed.

The order is not random: the first two steps decide where the secret lives, the next two limit who can read it, the last two concern what to do when the defenses fail. If you can do only one step today, do the sixth: knowing how to rotate a secret turns a leak from catastrophe into maintenance.

If a secret has leaked: the procedure

1. Revoke the secret immediately, from the panel of the service it gives access to (do not "wait to understand").
2. Generate a new secret and distribute it where it is needed (environment file, secrets manager).
3. Restart the services that use it, and check that they work.
4. Look for misuse: the logs of the service concerned in the hours between the leak and the revocation.
5. Remove the secret from the point of leak (the commit, the log, the page) after the revocation, remembering that removing it is not enough without step 1.
6. Note down what happened and how to prevent it from happening again (in the runbook, as this project does for every error solved).

The order matters: step 1 comes before step 5. Deleting the secret from the page without having revoked it leaves valid what others may already have copied, and step 4 tells you whether the leak has already had consequences.

Try it yourself: four guided exercises

Exercise 1: the three forms of a variable

In the terminal, reproduce the five lines of the block on the mechanism with sh -c 'echo "[$ZZ]"': without the variable, with ZZ=hello in front of the command, right after, after export ZZ=export and after unset ZZ. You should see [], [hello], [], [export], []. Then env -i /usr/bin/env | wc -l: in my test, 0 variables. What you learn: that variables go down to child processes and do not come back up, and that a program can work only thanks to a variable you forgot you had.

Exercise 2: the configuration that stops right away

Write the Settings class with a required field (api_token: SecretStr) and a numeric one (port: int). Start it without the token: you should see Field required with the name of the field. Set PORT=eight and observe unable to parse string as an integer. Then put API_TOKEN=abc in a .env, print Settings() and check that the token appears as SecretStr('**********'). Finally set API_TOKEN=other in the shell: you should see that the shell wins over the file. What you learn: the precedence among the sources and the value of failing at startup.

Exercise 3: where a secret really lives

Start a container with docker run -d --rm --name envt --env-file e.env python:3.12-slim sleep 20 (with API_TOKEN=abc123 in the file) and read docker inspect envt --format '{{range .Config.Env}}{{println .}}{{end}}'. Then build an image with ENV API_TOKEN=abc123 in the Dockerfile and observe docker history --no-trunc. Finally, on Linux, MIO=secret sleep 5 & and tr '\0' '\n' < /proc/$!/environ. You should see the value in plain text in all three points. What you learn: that an environment variable keeps a secret out of the code and out of Git, but does not make it invisible to whoever administers the machine.

Exercise 4: the secret with the newline

Create a file with printf 'tok123\n' > segreto, read it from Python with and without .strip() and print repr(...) of both. You should see 'tok123\n' and 'tok123'. Then write a small test that compares the value read with the expected one and check that it fails without .strip(). What you learn: one of the most frustrating causes of an authentication that "does not work", and how to make it impossible with a test.

What you can do now

You can keep configuration out of the code and read it from the environment, write a configuration class that stops at startup with a clear message when a value is missing or has the wrong type, and predict the precedence among environment, .env file and defaults. You can keep the real .env out of Git, document its names in a .env.example checked by a test, not bake secrets into images and not print them in the logs. And you know what to do when a secret leaks, in the right order. The next step is to understand what it means to put a program in a container: containers, images and isolation.

The same series also includes ML-Ops: UV, environments and dependencies that repeat identically, ML-Ops: testing a model with invariants, thresholds and data (previous article), ML-Ops: MLflow, tracking experiments instead of remembering them and ML-Ops: model registry, versions, aliases and promoting a model.