112 - ML-Ops: MLflow, tracking experiments instead of remembering them

MLflow is an open source tool (Apache 2.0 license) that records what happens when you train a model: with which parameters, with what result, producing which files. It is the answer to a question a notebook does not remember: "which experiment produced that number?". This article takes you from zero to four experiments recorded and compared with a query, and then to a parameter search with cross-validation. For the complete overview of the tool there is already the guide to experiment tracking with MLflow; here we build the case starting from the module's project.

All the Python listings were run twice, in a temporary folder outside the repository, with MLflow 3.18.0, scikit-learn 1.9.1, pandas 3.1.1 and Python 3.12, and the two runs gave the same output. I did not start MLflow's web server or the shared server with PostgreSQL (they require a listening port or external services): in the text they are declared schema to adapt. The data is synthetic and invented.

The words of tracking

Experiment tracking is the practice of recording, for each training, which inputs it was done with and what it produced: like keeping a lab notebook. An experiment is a container of trials on the same problem (here churn-baseline), a run is a single recorded execution, that is, one line of the notebook, with a unique identifier. Parameters are the inputs you choose before training (the regularization C, the seed, the number of rows) and, once recorded in a run, they are not changed; metrics are the outputs you measure (the AUC, the loss) and can be recorded several times with a step number (step). Artifacts are the files produced by the run: the model, a chart, a configuration. A tag is a free-text label, for example the name of the dataset.

It remains to understand where MLflow puts all this. The backend store holds the lines of the notebook (runs, parameters, metrics) in a database; the artifact store holds the attachments in a folder or in an object store: like the catalog and the shelves of a library. The tracking URI is the address that tells MLflow where the data lives (sqlite:///mlflow.db). The MLflow Tracking documentation describes precisely the backend store as the place that keeps the metadata of every run, and distinguishes runs (with parameters and metrics) from artifacts (the output files).

The life cycle of a run, and the final comparison:

In the drawing the order of the steps reflects that of the script we will see: parameters are noted before or during training, metrics and the model after, and the run closes by itself at the end of the with block. The comparison with search_runs happens outside the run, over all the ones already closed.

The analogy: the lab notebook

A serious chemist keeps a notebook: date, quantities of reagents, temperature, what was observed. Not out of bureaucracy: because three weeks later, faced with an unexpected result, they must be able to say "I did exactly this". The notebook is also what allows a colleague to repeat the experiment. MLflow is the lab notebook of ML, but filled in by the code, so it never forgets to write things down.

Why not a spreadsheet: three pains that tracking cures

For five experiments a spreadsheet is perfectly fine, and it is often what you start with. Structured tracking solves three pains that the spreadsheet solves badly, and which all show up after the tenth trial.

The pain of comparison. You have twenty rows, each with parameters and a result. Which one combines the best result with an acceptable cost? Which parameters have an effect and which do not? Sorting and filtering a spreadsheet works as long as the columns are always the same; as soon as an experiment adds a new parameter, the spreadsheet warps. A system that keeps each run as a record with arbitrary parameters and metrics makes querying uniform (search_runs).

The pain of reproduction. The spreadsheet row says "AUC 0.74, C=1". But with which dataset? Which seed? Which version of the code? Which libraries? If the row does not say, the number is not reproducible, even if another honest person wrote it. Tracking pushes you to record the context at the moment you know it, that is, during the run, not weeks later from memory.

The pain of communication. The colleague, or the you of six months from now, must understand what was done. A spreadsheet with twenty rows and names like "trial3_ok" cannot be understood; an experiment with named runs, tags and descriptions can. For whoever must decide whether the model can go to production, the MLflow interface is a dashboard that shows what was tried and what was chosen.

Preparing the environment

With UV (see uv and package management) you add MLflow to the project with uv add mlflow, and verify with uv run python -c "import mlflow; print(mlflow.__version__)" (schema commands, to adapt to your project). MLflow needs to know where to put the data. To begin, SQLite is enough, a local file (sqlite:///mlflow.db): no server to install.

A warning I verified with MLflow 3.18.0: the old folder backend (./mlruns, the one many tutorials still use) raises an exception by default. If an old tutorial does not work, this is probably why.

import mlflow
mlflow.set_tracking_uri("./mlruns")
try:
    with mlflow.start_run():
        pass
except Exception as e:
    print(type(e).__name__, "|", str(e))

# MlflowException | The filesystem tracking backend (e.g., './mlruns') is in maintenance mode
# and will not receive further updates. Please migrate to a database backend
# (e.g., 'sqlite:///mlflow.db') to access the latest MLflow features. [...]
# If the filesystem backend is required for your workflow, set `MLFLOW_ALLOW_FILE_STORE=true`
# to opt out of this exception.

The message (which I shortened) says two things: the folder backend is in maintenance, and there is a variable, MLFLOW_ALLOW_FILE_STORE=true, to postpone the choice. It is better to start directly from SQLite.

The test project: data and model

The examples use two modules of the churn project. I give them in minimal form, so that everything is runnable: make_dataset generates synthetic customers with tenure and complaints, train splits off a test set, trains a logistic regression and returns the model and the AUC.

# churn/data.py
import numpy as np
import pandas as pd

FEATURES = ["tenure", "complaints"]

def make_dataset(n: int = 800, seed: int = 0) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    tenure = rng.integers(0, 72, n)
    complaints = rng.integers(0, 8, n)
    logit = 0.3 - 0.03 * tenure + 0.25 * complaints
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "complaints": complaints, "churn": churn})

# churn/model.py
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

from churn.data import FEATURES

def train(df, seed: int = 0, C: float = 1.0):
    X_tr, X_te, y_tr, y_te = train_test_split(
        df[FEATURES], df["churn"], test_size=0.25, random_state=seed, stratify=df["churn"]
    )
    model = LogisticRegression(C=C, max_iter=500).fit(X_tr, y_tr)
    auc = roc_auc_score(y_te, model.predict_proba(X_te)[:, 1])
    return model, float(auc)

The seed fixes both the data generation (default_rng(seed)) and the split between training and test (random_state=seed): without it, the same code would give different numbers at each run, and comparing runs would become impossible. stratify keeps the proportion of customers who leave in both parts. C is the inverse of scikit-learn's regularization strength: small values constrain the coefficients more.

The first tracking script

# track.py
import mlflow
from churn.data import make_dataset
from churn.model import train

mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("churn-baseline")

df = make_dataset(n=800, seed=0)
for C in (0.01, 0.1, 1.0, 10.0):
    with mlflow.start_run(run_name=f"logreg-C{C}"):
        model, auc = train(df, seed=0, C=C)
        mlflow.log_params({"C": C, "seed": 0, "data_seed": 0, "n_rows": len(df)})
        mlflow.log_metric("auc", auc)
        mlflow.set_tag("dataset", "synthetic-v1")
        info = mlflow.sklearn.log_model(model, name="model")
        mlflow.set_tag("model_uri", info.model_uri)

What it does, line by line. set_experiment creates (or reuses) the experiment. with mlflow.start_run(...) opens a run and closes it at the end of the block, even if there is an error. log_params and log_metric note inputs and outputs: note that I also record data_seed, the seed of the data, because it is an input like the others (we will see why in the reproduction test). set_tag adds free labels: I note the name of the dataset, because a month from now "on which data?" will be the first question. mlflow.sklearn.log_model saves the model as an artifact with the information needed to reload it; in MLflow 3 the parameter is called name (in earlier versions it was artifact_path). Each log_model took about ten seconds on my machine: times change from one machine to another.

Comparing the experiments

import mlflow
mlflow.set_tracking_uri("sqlite:///mlflow.db")
runs = mlflow.search_runs(experiment_names=["churn-baseline"], order_by=["metrics.auc DESC"])
print(runs[["params.C", "metrics.auc"]].to_string())

#   params.C  metrics.auc
# 0     0.01     0.729467
# 1      0.1     0.728466
# 2     10.0     0.728165
# 3      1.0     0.728165

search_runs returns a pandas table with one row per run, sorted by descending AUC. Run twice, the outcome is identical. The table tells one useful thing and one warning: C = 1.0 and C = 10.0 give the exact same AUC (0.728165), while C = 0.01 is, in this example, slightly better than the others (0.729467). The difference between the first and the last is about a thousandth: with 200 customers in the test set, a distance this small does not tell a better model from chance. We will verify it below with cross-validation. Without the table you would have had to remember the numbers, or rerun everything.

The web interface

uv run mlflow server --backend-store-uri sqlite:///mlflow.db --port 5000
# schema to adapt, not executed: opens a listening port on your computer

Then open http://localhost:5000. In the interface you can sort runs by metric, select two or more and compare them with charts, open the artifacts. It is useful for exploring; for systematic analysis the script with search_runs remains more reproducible, because you can rerun and version it.

Something MLflow does on its own that is worth gold

If the project has a pyproject.toml and a uv.lock, MLflow notices and records the exact dependencies together with the model. I tried it in a folder with the two files (dependencies: mlflow, scikit-learn, pandas): during log_model the log lines Detected uv project: found uv.lock and pyproject.toml and Exported 92 dependencies via uv appeared, and the artifact's requirements.txt file had 91 lines. The number depends on your lockfile and on the date. The link with the article on UV is direct: the lockfile does not serve only to rebuild the development environment, it is archived next to the model, so that whoever loads it a year later knows which versions it was produced with (the trap of model versions is described in the article on the Dockerfile).

What to track, and what not

Avoid instead recording sensitive or personal data as parameters or artifacts, and do not record secrets (tokens, passwords): the tracking server is a shared archive, often with permissions wider than the code. The criterion for deciding is to ask, for each value, whether it serves to reproduce or understand the run: if so, it is recorded; if it contains something you would not want to see in a public log, it is not.

What was really saved: looking inside

A tool whose output you do not know remains a black box. After the four experiments I looked at what was on disk. The mlflow.db file is an ordinary SQLite database: you query it with Python's sqlite3 module. The tables are dozens, because MLflow also covers uses that are not needed here; the three that matter are runs, params and metrics. With the SQLite backend, the model files end up by default in the mlruns/ folder next to the script.

import sqlite3, pathlib
con = sqlite3.connect("mlflow.db")
for t in ("runs", "params", "metrics"):
    print(t, con.execute(f"select count(*) from {t}").fetchone()[0])
print("tables:", con.execute("select count(*) from sqlite_master where type='table'").fetchone()[0])
m = next(pathlib.Path("mlruns").rglob("MLmodel"))
print(sorted(p.name for p in m.parent.iterdir()))
print("requirements lines:", len((m.parent / "requirements.txt").read_text().splitlines()))

# runs 4
# params 16
# metrics 4
# tables: 71
# ['MLmodel', 'conda.yaml', 'model.skops', 'python_env.yaml', 'requirements.txt']
# requirements lines: 6

The numbers confirm the mental model: 4 runs and 4 metrics (one AUC per run), and 16 parameters, that is, 4 parameters (C, seed, data_seed, n_rows) for each of the 4 runs. You can query the database directly, and it is the right tool for an analysis the interface does not offer, but it is better to use MLflow's functions, which do not depend on the internal schema. In the model folder there are five files. MLmodel is a small text file with the "flavors" in which the model can be loaded: sklearn (as a scikit-learn object) and python_function (as a generic predict function, MLflow's universal format). conda.yaml and python_env.yaml describe the environment in two formats. requirements.txt lists the exact libraries at the time of saving: 6 lines without a UV project (mlflow, numpy, pandas, scikit-learn, scipy, skops), against the 91 of the test with uv.lock seen above.

A notable detail is that the model is saved as model.skops, a format of the skops project, and not as a pickle (the MLmodel file reports serialization_format: skops). The scikit-learn page on persistence explains why: Python's pickle can execute arbitrary code when it is loaded, while skops, unlike pickle-based formats, does not allow automatic execution of arbitrary code at load time. For an ML engineer it is a good habit to remember that a model file is code, not just data, and must be loaded only from sources you trust.

Parameters do not change; metrics can have a history

Two behaviors of tracking worth knowing, tested with MLflow 3.18.0. The first: recording the same parameter twice with different values in a run is an error. The second: metrics can be recorded several times with a step number, and MLflow keeps the series. In the same listing I also record two non-numeric attachments, a dictionary and a text.

import mlflow
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("trials")
with mlflow.start_run():
    mlflow.log_param("C", 1.0)
    try:
        mlflow.log_param("C", 2.0)
    except Exception as e:
        print(type(e).__name__, "|", str(e)[:100])
    for step, loss in enumerate([0.9, 0.6, 0.45]):
        mlflow.log_metric("loss", loss, step=step)
    mlflow.log_dict({"features": ["tenure", "complaints"]}, "config/features.json")
    mlflow.log_text("free note", "note.txt")

# MlflowException | Changing param values is not allowed. Param with key='C' was already logged with value='1.0' for run
# (the saved "loss" series has three points: 0.9, 0.6, 0.45)

The second log_param raises MlflowException with the message "Changing param values is not allowed": a parameter describes how the run was made, and rewriting it would falsify the notebook. If a value changes, it is another run. Metrics with step serve for models trained over several iterations (the "loss" that drops at every step): the interface draws the curve, while for a model trained in one shot, like our logistic regression, a final metric is enough. log_dict saves a dictionary as a JSON file at the given path and log_text a string as text: they are the simple way to attach what is not a number, such as the list of features, the report, or a mlflow.log_text(commit_hash, "git_commit.txt") to tie the run to the exact code.

Autolog: convenient, at a cost

MLflow can record parameters, metrics and model by itself for the most common libraries, with one line, mlflow.sklearn.autolog(), before training (I did not run it for this article). The advantage is speed; the limits are two. It records what it decides and can include more noise than you want, with many internal parameters. And it does not know your domain: the dataset name, the data version or the commit it does not invent. A good compromise is to use autolog to start quickly, plus a few explicit calls (set_tag("dataset", ...)) for what only you know.

Experiments that really repeat: the test

The promise of tracking is reproducibility. It must be verified, not believed. The listing reads from MLflow the parameters of the run with C = 1.0, retrains with those values and compares the metric:

import mlflow
from churn.data import make_dataset
from churn.model import train

mlflow.set_tracking_uri("sqlite:///mlflow.db")
r = mlflow.search_runs(experiment_names=["churn-baseline"],
                       filter_string="metrics.auc > 0.7 and params.C = '1.0'")
run = r.iloc[0]
C = float(run["params.C"]); seed = int(run["params.seed"])
n = int(run["params.n_rows"]); data_seed = int(run["params.data_seed"])
_, auc2 = train(make_dataset(n, seed=data_seed), seed=seed, C=C)
print("logged", run["metrics.auc"], "| rebuilt", auc2, "| equal:", auc2 == run["metrics.auc"])

# logged 0.7281650641025641 | rebuilt 0.7281650641025641 | equal: True

The reconstructed number is identical down to the last digit, because I recorded everything that was needed (parameters, seeds, data size) and because the code is deterministic. Two details. The first: filter_string is a small query language, with metrics.auc > 0.7 and params.C = '1.0'; parameters are strings (hence the quotes around '1.0') and must be converted back (float(...)) to reuse them. The second is the gap not to leave: if the script did not record the data seed (data_seed), you would have to remember it from memory, and the reproduction would fail without saying why; that is why it is among the parameters. Everything that influences the result goes among the parameters, including the origin of the data.

One measurement is not enough: record the mean and the deviation

A single AUC value depends on the split between training and test: with different splits it changes, and the comparison between C = 1.0 and C = 10.0 risks being a comparison of noise (the topic is covered in the article on testing models). A better practice is cross-validation: the dataset is split into 5 parts, you train 5 times using a different part each time for evaluation, and you get 5 AUC values, of which you record the mean and the standard deviation. The listing repeats the experiment on the four values of C with nested runs: a "parent" run for the search and four "child" runs, one for each C.

import mlflow
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score

from churn.data import FEATURES, make_dataset

mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("churn-cv")
df = make_dataset(n=800, seed=0)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

with mlflow.start_run(run_name="search-C") as parent:
    for C in (0.01, 0.1, 1.0, 10.0):
        with mlflow.start_run(run_name=f"C={C}", nested=True):
            scores = cross_val_score(LogisticRegression(C=C, max_iter=500),
                                     df[FEATURES], df["churn"], cv=cv, scoring="roc_auc")
            mlflow.log_param("C", C)
            mlflow.log_metrics({"auc_mean": scores.mean(), "auc_std": scores.std()})

children = mlflow.search_runs(experiment_names=["churn-cv"],
                            filter_string=f"tags.mlflow.parentRunId = '{parent.info.run_id}'",
                            order_by=["metrics.auc_mean DESC"])
print(children[["params.C", "metrics.auc_mean", "metrics.auc_std"]].round(4).to_string())
spread = children["metrics.auc_mean"].max() - children["metrics.auc_mean"].min()
print("gap between best and worst C:", round(spread, 4), "| typical auc_std:", round(children["metrics.auc_std"].mean(), 4))

#   params.C  metrics.auc_mean  metrics.auc_std
# 0     10.0            0.7364           0.0193
# 1      1.0            0.7364           0.0193
# 2      0.1            0.7358           0.0194
# 3     0.01            0.7356           0.0196
# gap between best and worst C: 0.0008 | typical auc_std: 0.0194

What it does: nested=True opens a run inside another, and MLflow records the link (tag mlflow.parentRunId), so in the interface the search appears as a single expandable entry and in the script the children are found again with a filter, which here returns four. cross_val_score returns the 5 scores; log_metrics records more than one with a single call.

Reading this table is the skill we want to train. The difference between the best value of C and the worst is 0.0008; the standard deviation of the individual folds is about 0.019, more than twenty times larger. In other words: among these four values of C there is no difference the data can tell apart from noise. Choosing C = 10 because it "has the highest mean" would be an arbitrary choice presented as a result; and the comparison in the previous script, where C = 0.01 seemed the best, is the example of what a single split can make you believe. The honest conclusion is that the four models are equivalent on this problem, and it is better to choose the simplest or the most stable (smaller C, more regularized). Without having recorded the standard deviation, you would have had a ranking and a false certainty. The numbers depend on the synthetic data and the seed: they are not a property of logistic regression.

Searching runs: the filter language

search_runs accepts a filter with a SQL-like syntax. I tried those in the table on the churn-baseline experiment (four runs), and the right-hand column reports how many runs they return:

Sorting is given with order_by=["metrics.auc_mean DESC"], and max_results limits the number. The result is a pandas table (DataFrame) with params.*, metrics.* and tags.* columns: you can compute whatever you want on it. The MLflow page on searching runs describes the complete syntax of the filters.

More people, one tracking server

With SQLite in a local file the notebook is yours. For a team you need three things: a server that shows everyone the same interface, a real database as backend (usually PostgreSQL) and a shared store for the artifacts (a network folder or an object store). The start line has the same shape as the local one, with different addresses:

mlflow server --backend-store-uri postgresql://user@host/mlflow \
              --artifacts-destination s3://artifacts-bucket \
              --host 0.0.0.0 --port 5000
# schema to adapt, not executed: requires a database and a store to connect to

The addresses are examples. With a shared server the rules of the article on environment variables and secrets come back: the server address (MLFLOW_TRACKING_URI) comes from an environment variable, and the credentials do not end up in the code. And the security questions come back: who can read the registered models? Who can promote an alias (the article on the model registry covers it)? The --host 0.0.0.0 option opens the server to the whole network: do it only behind authentication and a proxy.

The real case: training without tracking

This site's ML service retrains a model (an ensemble for a sports forecasting system) every Monday at 03:00 UTC with a systemd timer (OnCalendar=Mon *-*-* 03:00:00 UTC in infra/systemd/sport-intelligence-rolling-train.timer). The unit launches python -m app.sport_intelligence.train_advanced --catboost --output /var/lib/portfolio/sport_intelligence/model_v2.joblib. It does not use MLflow, and for a project with one person and a single model it is a defensible choice. But it shows where the cost lies: from the unit you read that the output always goes to the same path and that there is no step that preserves the previous version or its metrics (I did not verify other mechanisms on the server). If one week the model gets worse, the question "what was last week's like?" has no structured answer. It is exactly the pain that tracking solves, and the right moment to introduce it is when that pain shows up.

Trade-offs

• MLflow versus a spreadsheet: for five experiments, the spreadsheet is enough. MLflow wins from the first ten to twenty, when search, comparison and linked artifacts matter.
• SQLite versus a shared server: SQLite is perfect when working alone; for a team you need a tracking server with a real database (for example PostgreSQL) and a store for the artifacts.
• Tracking everything versus noise: too many metrics per run make the comparison unreadable. Few, chosen beforehand.

The three choices share the same underlying criterion: adopt the tool when the cost of not having it (comparisons from memory, non-reproducible results) exceeds the cost of running it. The thresholds "five" and "ten to twenty experiments" are a rule of thumb, not a measurement: they depend on how expensive it is to redo a trial and on how many people read the results.

How to organize experiments so they stay readable

One experiment per question: churn-baseline, churn-cv, churn-new-features; mixing different questions in the same experiment makes the comparison useless. Change one thing at a time, because if in a run you change parameter, data and features together you do not know what to attribute the difference to. Give runs speaking names (C=0.1, no-complaints, not trial7) and use a small vocabulary of consistent tags (dataset, commit, purpose): tags serve to filter, and a tag written in three ways does not filter. Use nested runs for parameter searches, record failed runs too with their status, because knowing that a configuration does not work is information, and every so often clean up the test runs: an archive full of noise discourages whoever consults it.

The opposite risk to the lack of tracking is excess: recording two hundred metrics per run "because you can", and then looking at none. The criterion is to choose beforehand a few metrics, the ones you will use to decide. Typical temptations: changing the main metric after seeing the results (p-hacking: choosing after the fact the number that confirms what you hoped), comparing runs made on different datasets as if they were equivalent, recording personal data among the artifacts, and using tracking as a substitute for reflection instead of as its support. Forgetting to close a run is another common error: always use the with block. And remember that the mlruns/ folders and the mlflow.db file are output, not code: they go in the .gitignore and in the .dockerignore.

A work routine in four moves

1. Before training: choose the name of the experiment and what you will vary (one parameter at a time).
2. During: with start_run, record parameters, seed, dataset, commit.
3. After: search_runs sorted by metric, and look at the numbers together with the variability across folds.
4. Decide and record the decision: a tag on the chosen run (mlflow.set_tag("chosen", "yes")) or, better, move on to the registry.

The order is that of a line of reasoning: the first move fixes the question, the second guarantees the data is recorded when you know it, the third keeps you from deciding on the first number you see, and only the fourth produces a tracked decision. Skipping the first leads to experiments that cannot be compared; skipping the third to choices based on noise.

From tracking to process

Tracking answers the question "what did I try and what came out". It does not yet answer "which model goes to production and how do I change my mind". That is why the next article adds the registry: the run you chose (after reading the table with the standard deviation) becomes a registered version of the model, with its label and its promotion rule. The common thread is that at every step identity is preserved: the experiment has an identifier, the model a version, the release a tag or a digest (see registries, tags and digests), the code a commit hash. If at any moment you cannot answer "which one?", the chain broke there. To version large data and models there is also the article on versioning datasets and models.

Try it yourself: four guided exercises

Exercise 1: four runs and a query

With this article's churn project (or a model of your own), run the tracking script with four values of C and then query the results with mlflow.search_runs(..., order_by=["metrics.auc DESC"]). You should see a sorted table (in my test: 0.7295 for C=0.01, 0.7285 for 0.1, 0.7282 for 1 and 10). If you like, also start the interface with uv run mlflow server --backend-store-uri sqlite:///mlflow.db --port 5000 and find the same runs by hand. What you learn: what gets recorded and the advantage of the query.

Exercise 2: looking inside

Open mlflow.db with Python's sqlite3 module and count the rows of the runs, params and metrics tables. Then list the mlruns/ folder and open the MLmodel and requirements.txt files of a saved model. You should see 4 runs, 16 parameters, 4 metrics and a requirements.txt with the exact versions. What you learn: that tracking is a database and some files, nothing magical.

Exercise 3: the reproduction test

Read from MLflow the parameters of a run, retrain with those values and compare the metric with ==. You should see equal: True. Then remove data_seed from the recorded parameters and try to reproduce: you will have to remember it from memory, and that is exactly the trouble to avoid. What you learn: that reproducibility is verified and that the typical gap is a forgotten parameter.

Exercise 4: mean and standard deviation

Redo the search on C with 5-fold cross-validation and nested runs (parent and children). Record auc_mean and auc_std. You should see that the difference between the best and the worst C is much smaller than the standard deviation (in my test 0.0008 against about 0.019). Write in one line the honest conclusion you draw from it. What you learn: not to choose a parameter for a difference the data cannot tell apart from noise.

Extra exercise: try recording the same parameter twice in the same run and read MLflow's error. Then change the value in a new run. The message tells you a rule of the system: a parameter describes how the run was made and is not rewritten.

What you can do now

You can record a training with MLflow on a SQLite database (experiment, run, parameters, metrics, tags, model), compare runs with search_runs and filter them with the filter syntax. You know what to track so that a number is reproducible (parameters, seeds of data and split, dataset, commit, dependencies) and how to verify it by rebuilding a run from what is recorded. You can read a comparison between parameters taking the standard deviation into account, and not pick a winner for a difference that noise explains. The next step is to give the chosen model a version and a name: model registry, versions and aliases.

The same series also includes ML-Ops: from notebook to service.