106 - ML-Ops: pytest, tests that let you sleep at night

pytest is the most widespread testing framework in Python: you write functions whose names start with test_, inside them you use the assert statement, and it finds them, runs them and tells you which ones fail and why. A test is an automatic check that answers a question: "does the code still do what I thought?". In this lesson, with pytest 9.2, you will learn the five moves that cover almost everything: assert, fixtures, parametrization, expected exceptions and markers. The next lesson applies them to models. It assumes an environment created as in UV, environments and dependencies.

The words of testing

A test is a small function that runs a part of your code and checks that the result is the expected one, like the cook's taste: a sample, not the whole lunch. The assertion (the assert statement) is the claim "this must be true": if it is false, the test fails. The test suite is the set of all the project's tests, run together, and a regression test protects a behavior that used to work, so that a future change does not break it (the "regression" is precisely going backwards). A fixture is the preparation of what a test requires (data, a file, a connection), repeatable and clean like setting the table before every dinner and clearing it afterwards. Parametrization runs the same test with several sets of values, a marker is a label on a test (for example slow) with which you can select or exclude it.

A mock (a test double) is a fake object that replaces a real part (a database, a network service) so as to test the rest without depending on it, like the dummy in crash tests. The exit code is the number with which a program says how it went (0 = fine): pytest returns 0 if everything passes and other numbers if something is wrong, and it is what CI reads.

The analogy: the pre-takeoff checklist

An experienced pilot runs the checklist before the flight, every time, even though they have flown that route a thousand times. Not because they do not trust themselves, but because the checklist is more reliable than memory and costs two minutes. Tests are your code's checklist: every time you change something, in two seconds you know whether you have broken something that used to work.

The first test

The examples that follow were run on 2026-11-12 with Python 3.12.13, pytest 9.2.0 and pytest-cov installed by UV in a test project called pulizia (uv init --bare --python 3.12 pulizia, uv add --dev pytest pytest-cov), twice in different folders with the same results. Names of functions, files and variables are kept as in the Italian original (normalizza_eta is "normalize age", pulizia is "cleaning"), so that the outputs match exactly. Let us take a function that cleans the raw age read from a file, typically dirty (spaces, decimal comma, impossible values):

# src/pulizia/eta.py
def normalizza_eta(valore: object) -> int:
    """Converts a raw value into a valid integer age (0-120)."""
    if valore is None or str(valore).strip() == "":
        raise ValueError("age missing")
    eta = int(float(str(valore).replace(",", ".")))
    if not 0 <= eta <= 120:
        raise ValueError(f"age out of range: {eta}")
    return eta

And the test, in tests/test_eta.py:

from pulizia.eta import normalizza_eta

def test_intero_semplice():
    assert normalizza_eta(42) == 42

No classes, no special methods: a function and an assert. If the condition is false, the test fails. You launch it with uv run pytest. In pyproject.toml you declare once where the tests are and where the code lives:

[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["src"]

testpaths tells pytest where to look when you launch it without arguments; pythonpath adds src to the import paths, and is what lets from pulizia.eta import ... work with the src/ folder layout. If you omit pythonpath the default does not include src: removing the line I got ModuleNotFoundError: No module named 'pulizia' and 1 error during collection.

Parametrization: one test, many cases

Errors are born at the edges: minimum values, maximum values, odd formats. Instead of writing ten almost identical tests, you write one and pass it the cases:

import pytest

@pytest.mark.parametrize(
    "grezzo, atteso",
    [("42", 42), (" 7 ", 7), ("33,9", 33), (0, 0), (120, 120)],
)
def test_formati_validi(grezzo, atteso):
    assert normalizza_eta(grezzo) == atteso

pytest runs the test five times and reports them separately, with the case in the name (test_formati_validi[ 7 -7]). When one fails you know exactly which. Note the "33,9" case that gives 33: the function truncates, it does not round. The test documents the decision, and if someone changes it, the test says so.

Expected exceptions

Even the correct failure is a behavior to test: an age of 200 years must produce an error, not pass silently.

@pytest.mark.parametrize("grezzo", [None, "", "   ", -1, 121, "999"])
def test_valori_non_validi_sollevano_errore(grezzo):
    with pytest.raises(ValueError):
        normalizza_eta(grezzo)

def test_messaggio_di_errore_e_utile():
    with pytest.raises(ValueError, match="out of range"):
        normalizza_eta(200)

match= also checks the message, as a regular expression searched in the error text: an error that says what went wrong is worth more than a mute one.

Fixtures: preparing the data once, cleanly

When several tests need the same starting data, you prepare it in a fixture and the test requests it simply by naming it as an argument. It goes in tests/conftest.py, a file that pytest reads by itself:

import pytest

@pytest.fixture
def clienti():
    """Small, hand-built data: each test receives a fresh copy."""
    return [{"id": 1, "eta": "34"}, {"id": 2, "eta": "61,5"}]

def test_tutti_i_clienti_hanno_eta_valida(clienti):
    assert [normalizza_eta(c["eta"]) for c in clienti] == [34, 61]

By default a fixture is recreated for every test: this way no test dirties another's data. If the preparation is expensive (training a model, for example) you can widen the scope, with @pytest.fixture(scope="module"): recreated once per file. It is a compromise between speed and isolation, and we will use it in the lesson on how to test a model.

pytest offers ready-made fixtures. The most useful are tmp_path (a temporary folder that disappears at the end of the test, to test file writes) and monkeypatch (changes environment variables or attributes only for the duration of the test, then restores):

def test_variabile_ambiente(monkeypatch):
    monkeypatch.setenv("ETA_MAX", "100")
    import os
    assert os.environ["ETA_MAX"] == "100"

You will meet monkeypatch again in the lesson on environment variables, to test configuration without touching the real environment.

Markers: separating fast tests from slow ones

A slow test that runs on every save is soon ignored. A marker is a label, and you declare it in pyproject.toml so that pytest warns you if you get the name wrong:

[tool.pytest.ini_options]
markers = ["slow: slow tests, excluded by default with -m 'not slow'"]

Then you apply the marker to the slow test with the @pytest.mark.slow decorator, which changes nothing in the body of the test but labels it:

@pytest.mark.slow
def test_scansione_lunga():
    assert sum(range(1_000_000)) > 0

Finally you choose on the command line whether to include or exclude the labeled tests; the counts below come from the test project:

uv run pytest -q                  # everything
# 20 passed
uv run pytest -q -m "not slow"    # without the slow ones
# 19 passed, 1 deselected

The counts are those of the test project with the complete test file (20 tests); the marker text says "excluded by default", but the exclusion happens only if you pass -m 'not slow' yourself (or put it in addopts): the marker alone just labels. In later lessons we will use a marker for the checks on the model.

Reading a red test

I deliberately changed the expected value of the " 7 " case from 7 to 8. This is what pytest shows:

__________________________ test_formati_validi[ 7 -8] __________________________

grezzo = ' 7 ', atteso = 8

    @pytest.mark.parametrize(
        "grezzo, atteso",
        [("42", 42), (" 7 ", 8), ("33,9", 33), (0, 0), (120, 120)],
    )
    def test_formati_validi(grezzo, atteso):
>       assert normalizza_eta(grezzo) == atteso
E       AssertionError: assert 7 == 8
E        +  where 7 = normalizza_eta(' 7 ')

tests/test_eta.py:16: AssertionError
=========================== short test summary info ============================
FAILED tests/test_eta.py::test_formati_validi[ 7 -8] - AssertionError: assert...
1 failed, 19 passed

You read it from the bottom: which test (FAILED ...), with which values (grezzo = ' 7 '), which line (>), what it expected and what it got (assert 7 == 8). Before changing the code, always ask yourself: is the code wrong or is the test wrong? In this case it was the test.

What good tests look like

Five qualities set a good test apart:

• Small: one thing per test, with a name that says which (test_valori_non_validi_sollevano_errore).
• Fast: in the order of milliseconds, because if they are slow people stop running them.
• Independent: the order does not matter and none depends on another's result.
• Deterministic: same outcome on every run, with a fixed seed in random data.
• On behaviors, not on lines: test what the function does, not how it does it.

The first four protect trust in the suite: a slow test, one that depends on another or that sometimes fails for no reason is soon ignored, and an ignored suite protects nothing. The last protects the code: if the tests describe the behavior you can rewrite the inside without rewriting the tests.

Traps

The first is tests that do not run: a file called check_extra.py or a function without the test_ prefix is ignored, and the suite stays green. Look at the number of tests run, not just the color; in this site's project some frontend specs are excluded from the configuration and the suite stays green without having run them, which is why the count is worth as much as the outcome. The second is tests that always pass, for example an assert on something that is always true: try breaking the code on purpose and check that the test turns red. The third is modified shared data: a fixture with a wide scope that tests modify produces failures that depend on the order. The fourth is testing too much: tests that replicate the implementation line by line break at every refactoring without protecting anything.

How pytest finds tests: discovery

pytest does not need to be "configured" to know what to run: it follows naming conventions. It looks for test_*.py or *_test.py files, and inside them for functions that start with test_ (and Test* classes). What does not respect the name is not collected, without any warning: it is the "tests that do not run" trap. To see what it collected without running it:

uv run pytest --collect-only -q
# tests/test_eta.py::test_intero_semplice
# tests/test_eta.py::test_formati_validi[42-42]
# tests/test_eta.py::test_formati_validi[ 7 -7]
# ...
# 20 tests collected in 0.04s

It prints an identifier for each test: file, function and, in square brackets, the parametrized case. Before trusting a green suite, check that the list contains what you expect. In the test I copied a test file under the name check_extra.py: uv run pytest -q kept saying 22 passed, before and after, while naming the file by hand (uv run pytest -q tests/check_extra.py) ran its test (1 passed).

The commands for daily work

I tried these options on the test project:

A typical work cycle: write the test, pytest -x to see the first failure, fix, pytest --lf to retry only that one, then the full pytest before the commit.

A telling detail: running pytest -k "nessunotest", which selects no test, the exit code is 5 (not 0, not 1): "no tests collected" (measured with echo $?). It is a failure, rightly, but some scripts treat it as a success. If a CI is configured to ignore 5, an empty suite passes for green.

Comparing decimal numbers: pytest.approx

Floating-point numbers are not exact: in Python 0.1 + 0.2 == 0.3 is False, because the computer represents those values in binary with a minimal approximation. In ML, where every result is a decimal, a comparison with == fails for reasons that have nothing to do with an error. You use pytest.approx:

def test_float_con_approx():
    assert 0.1 + 0.2 != 0.3                       # floats are not exact
    assert 0.1 + 0.2 == pytest.approx(0.3)        # comparison with tolerance

The first assert shows the phenomenon (it passes: they really are different), the second the remedy. By default the relative tolerance is 1e-6, one millionth of the expected value (pytest.approx documentation); it can be adjusted with pytest.approx(0.3, abs=1e-3) when a wider threshold is needed. You will meet it again in model tests.

Temporary files: tmp_path

A test that writes to disk must not dirty your project nor depend on files left by a previous run. tmp_path is a fixture that provides a new, empty folder for every test, later cleaned up by pytest according to its own retention policy:

def test_scrittura_su_file_temporaneo(tmp_path):
    destinazione = tmp_path / "risultati.json"
    destinazione.write_text(json.dumps({"auc": 0.71}))
    assert json.loads(destinazione.read_text())["auc"] == pytest.approx(0.71)

The test requests the fixture by writing its name as a parameter (you also need import json at the top of the file); pytest creates it and passes it in. The object is a standard-library Path, so / composes paths.

Fixtures with setup and cleanup: yield

When a fixture opens something (a file, a connection), it must also close it. You use yield: what comes before is the setup, what comes after runs at the end of the test, even if the test failed:

@pytest.fixture
def risorsa():
    eventi = ["open"]
    yield eventi             # the test runs here, receiving "eventi"
    eventi.append("close")   # teardown: always, after the test

def test_fixture_con_yield(risorsa):
    assert risorsa == ["open"]

Inside the test the list contains only "open"; the addition of "close" happens afterwards, which is why you cannot check it from the test itself. In a real fixture, in place of the list there would be a connection and in place of the append its closing.

Replacing a real piece: the mock

A test must be fast and deterministic, so it should not call a network, a database or a gigabyte-sized model. You replace the slow part with a double:

from unittest.mock import Mock

def test_doppio_con_mock():
    servizio = Mock()
    servizio.leggi.return_value = " 41 "
    assert normalizza_eta(servizio.leggi()) == 41
    servizio.leggi.assert_called_once()

Mock() creates an object that accepts any call; return_value sets what the leggi ("read") method returns; the last line checks that it was called exactly once. The danger of the mock is forgetting to also test the real thing: doubles say that your code uses the interface well, not that the interface works. A green test on a double does not replace a trial, at least once, against the real thing.

A real example from this site's ML service

The file ml-service/tests/test_cache_redis_down.py checks that the service does not freeze when Redis (the cache) does not respond. It builds a fake Redis that raises a connection error and checks that the service degrades gracefully. The test is written with the comments Arrange, Act, Assert, which make the three parts of a good test explicit; here it is abbreviated (in the file it sits inside a class) and the comments are expanded:

@pytest.mark.asyncio
async def test_get_returns_none_when_redis_raises():
    # Arrange: set up the conditions (a Redis that raises an error)
    redis = AsyncMock()
    redis.get = AsyncMock(side_effect=ConnectionError("Redis connection refused"))
    cache = CacheService(redis=redis)

    # Act: perform the action under test
    result = await cache.get("any query")

    # Assert: check the result
    assert result is None

AsyncMock is the version of Mock for asynchronous functions; side_effect=ConnectionError(...) makes it raise an error instead of returning a value. The test says: "if the cache is down, get returns None and does not propagate the error". It is a behavior to specify for a production service, and without the mock you would have to switch Redis off to try it. Arrange builds the data and the doubles, Act calls one thing only, Assert checks the outcome. If a test has more than one "Act", they are probably two tests; if the Arrange is very long, a fixture would make it more readable; if the Assert checks ten things, when it fails you will not know which. This block was not run: it requires the repository's ml-service and its dependencies, and the repository's tests must not be launched from this work; I only verified that the file exists and contains the test. The ml-service/tests/ folder has 64 test_*.py files at the first level (ls tests/test_*.py | wc -l, 2026-11-12).

The why: the fear of touching the code

Whoever works on code without tests knows a precise feeling: the fear of changing something that works. Every change is a bet, because there is no quick way to know what broke. The consequence is that the code stops evolving: names are not improved, functions are not simplified, libraries are not updated, because "better not touch it". In machine learning the fear is even more justified: a change to a data-cleaning function can worsen the model without any error (the silent degradation discussed in the module introduction). Tests are the only honest answer to this fear, because they turn it from a state of mind into information: "I changed this and these 21 checks say the rest behaves as before".

There is a second, less obvious reason: tests are documentation that does not age. A comment can say something false after six months; a failing test cannot. The lesson's test "33,9" -> 33 says in executable form that the function truncates and does not round: whoever changes that behavior is forced to read and decide.

Which tests to write: the pyramid

Not all tests are equal. Three levels, from the smallest to the largest, often drawn as a pyramid because you need many at the bottom and few at the top:

The pyramid shape has an economic reason: unit tests are fast, stable and tell you where the problem is; end-to-end ones are slow, more fragile and tell you only that there is a problem. A suite made only of end-to-end tests is slow and frustrating; one made only of unit tests does not discover defects at the joins between the parts. This site's ML service has unit tests (like the cache ones), infrastructure tests that require external services running, and marked evaluation checks, and it enforces an 80% coverage threshold only on one part (the tax module, with a separate pytest configuration file, as the comment in its pyproject.toml says): a threshold applied where the risk is greatest, not indiscriminately.

Writing the test first: the red, green, refactor cycle

TDD (test-driven development) reverses the order: first you write the test that describes the desired behavior, you see it fail (red), then you write the minimal code to make it pass (green), then you clean up (refactor) with the safety net of the tests. It is not a dogma, it is a way of thinking: it forces you to define what you want before how. A real example with our function. I want a value like "n/d" (Italian shorthand for "not available") to produce a clear error. I write the test first:

def test_testo_non_disponibile_e_un_errore_chiaro():
    with pytest.raises(ValueError, match="missing"):
        normalizza_eta("n/d")

I run it, and it is red, but in an instructive way:

E   AssertionError: Regex pattern did not match.
E     Expected regex: 'missing'
E     Actual message: "could not convert string to float: 'n/d'"
1 failed, 20 passed

The code did not crash silently: it already raised a ValueError, but with a message incomprehensible to whoever reads the logs ("could not convert string to float"). The test made visible a defect I had not been looking for. Then I write the minimal code: I treat "n/d" (and a few variants) as a missing value.

if valore is None or str(valore).strip().lower() in ("", "n/d", "nan", "none"):
    raise ValueError("age missing")

I rerun: green (21 passed). Then the refactor phase: with the suite green I can rename, move, simplify, knowing that if I break something I will see it immediately. Note the important detail: the test was seen failing first. A test never seen red might verify nothing; the lesson on testing models calls it "checking the check".

Coverage, with open eyes

Coverage measures which lines of code were executed by the tests. With the pytest-cov plugin:

uv add --dev pytest-cov
uv run pytest -q --cov=pulizia --cov-report=term-missing
# Name                      Stmts   Miss  Cover   Missing
# -------------------------------------------------------
# src/pulizia/__init__.py       0      0   100%
# src/pulizia/eta.py            7      0   100%
# -------------------------------------------------------
# TOTAL                         7      0   100%

Columns: Stmts the statements, Miss those never executed, Cover the percentage, Missing the uncovered line numbers. The 100% here says that every line of eta.py was executed at least once, not that every behavior is verified. To catch more there is branch coverage (--cov-branch): it also counts whether each if was seen both true and false. I tried it with a three-band age function and a single test (fascia_eta(30)):

# src/pulizia/fascia.py
def fascia_eta(eta: int) -> str:
    if eta < 18:
        return "minorenne"
    if eta < 65:
        return "adulto"
    return "senior"

# uv run pytest -q --cov=pulizia --cov-branch --cov-report=term-missing
# src/pulizia/fascia.py     6   2   4   2   60%   3, 6

The report indicates lines 3 and 6 not executed (the "minor" and "senior" cases, minorenne and senior in the code): the test covers one path out of three. This is the correct way to use it: coverage as a map of blind spots, telling you where to look, not as a number to inflate. A 100% coverage with tests without assertions is worse than 70% with serious tests, because it gives confidence without deserving it. The comment in the pyproject.toml of this site's ML service shows the right maturity: it reports the measured baseline coverage (57%, with 33 failing infrastructure tests because the services are missing) and sets the 80% threshold only where it is sustainable. Declaring the starting point is better than pretending a number.

Debugging a red test

When a test fails, the order of questions matters. First read the message from the bottom: which test, which assertion, which values. Then ask whether it is the code or the test that is wrong. If it is not obvious, show more context with the -l option (--showlocals), which adds the local variables to the report; for parametrized tests pytest already shows the parameter values (grezzo = ' 7 ', atteso = 8) by itself, as in the report above, so -l is mainly for variables created inside the function. To isolate, pytest -x -k test_name runs only that one and stops. If you need to investigate inside, pytest --pdb opens the Python debugger at the point of failure (I did not try it here), where you can print variables and execute expressions. After the fix rerun the whole suite, because a fix can break something else; and if the test fails only sometimes you have a problem of instability, not of logic.

The real case: the suite is never "green", and the number counts

The project's working rules contain a note about the frontend tests that is a small lesson in methodological honesty. In short: the suite is not green, there are pre-existing reds cataloged in an inventory; the criterion for judging a change is not "zero reds" but the delta (no new red test compared to before the change). And there is a trap: some test files are excluded from the configuration, so they never run, and the suite stays green without having tried them; this is why you check the number of tests run, not just the color. Another: launching the tests with a tool different from the authorized one produces mass false failures.

You draw three habits valid for any project, yours too. Measure the delta and not the absolute when there is historical debt: before modifying, run the tests and save the list of failures, afterwards compare. Count the tests: "50 passed" before and "30 passed" after is a problem even if there is no red. And use the tool the project indicates to run the suite, because an "almost identical" command can give different results for reasons unrelated to the code.

A small daily workflow with tests

1. Before modifying: uv run pytest -q and note the result (how many tests, how many reds).
2. Write or update a test for the behavior you want to obtain; see it red.
3. Write the code; see it green.
4. Run the whole suite and check the delta against step 1.
5. Check the coverage of the files you touched (--cov) to see whether you left untested branches.
6. Commit code and tests together (as in Git): this way git bisect always has consistent commits.

There is a reason for the order of every step: step 1 gives the reference without which step 4 makes no sense, 2 before 3 guarantees having seen the test fail, and 6 closes the loop so that the repository's history tells tests and code in the same commit.

Try it yourself: four guided exercises

Exercise 1: the first suite

Create a UV project (uv init --bare --python 3.12 pulizia, uv add --dev pytest), write the lesson's normalizza_eta function and the tests for three valid inputs and three invalid ones. Declare testpaths and pythonpath in pyproject.toml. You should see N passed with uv run pytest -q. What you learn: the minimal setup, and that without pythonpath = ["src"] a layout with a src/ folder gives ModuleNotFoundError (I reproduced it).

Exercise 2: the red, green cycle

Write a test for a behavior the code does not have yet (for example: the age as a string with a unit, "34 anni", must give 34). Run it and see it fail, reading the message from the bottom. Then write the minimal code that makes it pass. What you learn: to use the test as a specification; and, by observing the red message, to understand whether the defect is in the code or in the test.

Exercise 3: the test that does not run

Copy a test file giving it a name pytest does not recognize (check_extra.py instead of test_extra.py) and rerun uv run pytest -q. You should see that the number of tests run does not change (in my test, 22 before and 22 after): the file is silently ignored. Then run uv run pytest -q tests/check_extra.py naming the file by hand: this time it runs. Use uv run pytest --collect-only -q to see what gets collected. What you learn: to distrust a green suite and always check the number of tests.

Exercise 4: branch coverage

Write a function with three branches (for example one that assigns an age band: under 18, between 18 and 64, 65 or more) and a single test. Run uv add --dev pytest-cov and uv run pytest --cov=pulizia --cov-branch --cov-report=term-missing. You should see a partial coverage with the list of lines not executed (in my test, 60% with lines 3 and 6 missing). Add the missing tests with parametrize until everything is covered, but then ask yourself: do the tests verify the right bands, or do they merely run the code? What you learn: coverage as a map of blind spots, not as a goal.

A fifth exercise, for those who already have a project: pick the function you fear breaking most and write the three tests that would let you sleep easy. They are usually: a normal case, an edge case and an invalid input.

What you can do now

You can write a test as a test_... function with an assert and launch it with uv run pytest. You can use parametrize for edge cases, pytest.raises for expected errors, fixtures for starting data and markers to separate the slow ones. You can read a red test from the bottom (which one, with which values, expected versus obtained), see a test fail before trusting that it passes, and check how many tests run, not just whether they are green. The next lesson brings all this to models: testing a model with invariants, thresholds and data.

The same series also includes ML-Ops: using open source well, from licenses to contributing and ML-Ops: deployment, where the model runs and how to choose.