113 - ML-Ops: model registry, versions, aliases and promoting a model

A tracked experiment (see MLflow, tracking experiments) tells you how a trial went. A model registry answers a different question: which model is the one in use, and how do I change it without doing damage? It is a catalogue of models with a name, version numbers, metadata and aliases, that is, movable labels such as champion. With MLflow 3.18 you register the best model of an experiment, make a service load it by logical name rather than by file path, promote a new one with a written rule, and go back by moving a label.

All the Python listings in this article were run twice, in a temporary folder outside the repository, with MLflow 3.18.0, scikit-learn 1.9.1 and Python 3.12, and the two runs gave the same output (only the m-... identifiers that MLflow assigns to models change). They start from the same four experiments and the same synthetic data as the previous article, so the numbers are consistent with those. No MLflow server was started.

The words of the model registry

Two terms from the tracking article are needed here: the run, one recorded training execution, and the artifact, a file produced by a run, such as the saved model. The Model Registry documentation adds a small vocabulary, which I have summarised here:

• Registered model: a logical name (churn-classifier) under which the registry gathers all the versions. Like the title of a book that has several editions.
• Version: a numbered edition (1, 2, 3...). Every new registration under the same name increments the number; it does not overwrite the previous one. It is the stable identifier, the counterpart for models of the digest discussed in the article on registries, tags and digests.
• Alias: a named, modifiable reference that points to a single version. Like "the recommended edition" in a bookshop: today it is the third, tomorrow the fourth, while the editions themselves do not change.
• Champion and challenger: the two usual roles. The champion is the version in service; the challenger is the candidate being evaluated to replace it.
• Promotion and rollback: the first moves the champion alias onto a new version, the second puts it back on the previous one.
• Model URI: the address of a model, in the form models:/name@alias or models:/name/version.
• Lineage: the chain that links a version to the run that produced it, and that run to the data and the code.

The thread that holds the entries together is the distinction between what does not change and what moves: name and version number identify, the alias decides. The last two entries serve to answer "where does this model come from?" later on, and to load it without ambiguity. One entry is missing on purpose: stages (Staging, Production, Archived), the old MLflow system with fixed states. The documentation on the workflow states that since version 2.9.0 they are deprecated and will be removed in a future major version, so new code uses aliases.

Fixed versions and movable aliases in a drawing. The AUC values are those of the two models we will register further on:

The analogy: the duty roster

In a hospital ward there is the list of doctors (the people, each with their own qualifications) and the duty roster ("on call tonight: Dr. Bianchi"). If Bianchi falls ill, nobody rewrites the CVs: the name on the roster is changed. The registry works the same way. The versions are the people: numbered, with their credentials (metrics, dataset, dependencies). The alias is the shift: champion says which version serves the traffic right now.

Why a file is not enough

The naive solution is a model.joblib file in a folder, which every training run overwrites. A service that always saves the model to the same path and retrains it on a fixed schedule (say every Monday) works like this, and it works as long as everything goes well. When something goes wrong, four limits emerge: you cannot go back to last week because the old file no longer exists; you do not know which metric the file in use had; there is no point at which someone decides "this goes to production"; and a name like model_v2 indicates the generation of the model, not the individual training run. The registry exists to remove these four limits with a few calls.

Registering a model

We start from the four experiments of the previous article (churn-baseline, with C equal to 0.01, 0.1, 1.0 and 10.0). The script takes the run with the best AUC, registers its model under the name churn-classifier, records the AUC as a tag of the version and assigns the champion alias:

# register.py
import mlflow
from mlflow import MlflowClient

mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()

best = mlflow.search_runs(
    experiment_names=["churn-baseline"], order_by=["metrics.auc DESC"], max_results=1
).iloc[0]
mv = mlflow.register_model(best["tags.model_uri"], "churn-classifier")
client.set_model_version_tag("churn-classifier", mv.version, "auc", f'{best["metrics.auc"]:.4f}')
client.set_registered_model_alias("churn-classifier", "champion", mv.version)
print("version", mv.version, "-> alias champion", "| C =", best["params.C"])

# Created version '1' of model 'churn-classifier'.
# version 1 -> alias champion | C = 0.01

Three steps. register_model creates version 1 of the registered model; the auc tag records the quality on the version, so whoever consults the registry does not have to go back to the runs; set_registered_model_alias designates the version as the one in use. Note that version 1 is not the one with the highest C but the one with C = 0.01: it was the best by AUC, even though, as the previous article shows, the difference from the others is within the noise. The registry catalogues what you tell it to catalogue; it does not judge whether the choice makes sense.

Why best["tags.model_uri"] and not runs:/<id>/model? In the previous article I recorded the URI of the saved model as a tag (models:/m-...). I also tried the runs:/<id>/model form: MLflow 3.18 registers the model anyway, but prints a warning, "Run ... has no artifacts at artifact path 'model', registering model based on models:/m-... instead". It works, but it is a compatibility path: in MLflow 3 models are entities in their own right (logged model) with their own identifier, and it is better to refer to that.

Aliases, not stages

Aliases are arbitrary labels that you choose, each pointing to a single version. You can have champion and challenger at the same time, or one alias per environment. I register as a candidate the run with C = 10.0 (third by AUC, tied with C = 1.0) and give it the challenger alias:

import mlflow
from mlflow import MlflowClient

mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()
NAME = "churn-classifier"

run = mlflow.search_runs(experiment_names=["churn-baseline"], filter_string="params.C = '10.0'").iloc[0]
mv = mlflow.register_model(run["tags.model_uri"], NAME)
client.set_model_version_tag(NAME, mv.version, "auc", f'{run["metrics.auc"]:.4f}')
client.set_registered_model_alias(NAME, "challenger", mv.version)
for alias in ("champion", "challenger"):
    print(alias, "->", client.get_model_version_by_alias(NAME, alias).version)

# champion -> 1
# challenger -> 2

The script looks for the run with a filter (params.C = '10.0', parameters are strings), registers its model, which becomes version 2, and then prints where the two aliases point. An alias can point to only one version, but a version can have several aliases: neither is an error.

Loading by alias and promoting: the service does not know the number

The service code says "give me the champion", not "give me version 1". To change the model in service you touch neither the code nor rebuild the image (see the article on the Dockerfile): you move the alias and restart the service, which reloads it at startup. Updating the model and updating the software become two operations with two different rhythms. The script below loads a fixed customer (3 months of tenure, 4 complaints) by alias, then promotes version 2 and finally goes back:

import mlflow
import pandas as pd
from mlflow import MlflowClient

mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()
NAME = "churn-classifier"
cliente = pd.DataFrame({"tenure": [3], "complaints": [4]})

def risposta():
    model = mlflow.sklearn.load_model(f"models:/{NAME}@champion")
    return round(float(model.predict_proba(cliente)[0, 1]), 4)

print("before         :", risposta())
client.set_registered_model_alias(NAME, "champion", "2")   # promozione
print("after promotion:", risposta(), "| champion ->", client.get_model_version_by_alias(NAME, "champion").version)
client.set_registered_model_alias(NAME, "champion", "1")   # rollback
print("after rollback :", risposta(), "| champion ->", client.get_model_version_by_alias(NAME, "champion").version)

# before         : 0.7993
# after promotion: 0.8051 | champion -> 2
# after rollback : 0.7993 | champion -> 1

The function risposta (Italian for "answer", an identifier kept as in the listing) resolves the alias on every call, which is why after the promotion the same request gives a different probability (0.8051 against 0.7993): the model in service is another one, and the code has not changed. The rollback is the same line with the previous number: no rebuild, no file to recover. A process that is already running, however, does not see the move by itself: in the script the model is reloaded on purpose at every call, while a real service does it at startup and therefore has to be restarted. It is the same principle as the immutable identifier (the version) plus a label that moves (the alias).

A promotion rule written in code

Promoting "because it looks better" is the way to get a worse model into production. The decision has to be written down. The function compares the auc tag of the candidate with that of the champion and moves the alias only if the candidate beats it by at least a margin:

import mlflow
from mlflow import MlflowClient

mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()

NAME = "churn-classifier"
MARGINE = 0.01   # to be chosen from observed data, as for the variability between splits

def auc_of(client, version):
    return float(client.get_model_version(NAME, version).tags["auc"])

def promote_if_better(client, candidate):
    champion = client.get_model_version_by_alias(NAME, "champion").version
    if auc_of(client, candidate) >= auc_of(client, champion) + MARGINE:
        client.set_registered_model_alias(NAME, "champion", candidate)
        return True
    return False

print("promoted v2:", promote_if_better(client, "2"))
print("champion ->", client.get_model_version_by_alias(NAME, "champion").version)

# promoted v2: False
# champion -> 1

Trying to promote version 2 (AUC 0.7282) against version 1 (0.7295), the result is False and champion stays on version 1. The margin is not a detail. The previous article measured, with cross-validation on the same data, a standard deviation of the AUC between folds of about 0.019: a candidate that beats the champion by 0.002 on a single split is not better, it is lucky. The margin of 0.01 here is a teaching value, and strictly speaking it is still inside that noise: this is why further on the comparison moves to a fixed, larger evaluation set.

What is worth saving on every version

On every version record the main metric and the evaluation dataset it was measured on, the identity of the training data and the commit hash (see Git, version your work), who promoted it, when and why, in a free-text tag. You do not need the dependencies by hand: MLflow stores them by itself, lockfile included, if the project uses UV (uv and package management). The criterion for each value is to ask yourself whether, six months from now, faced with a model that behaves badly, you will need it to understand where it comes from.

A trap seen in testing: where did the aliases go?

Asking for the list of versions with search_model_versions, in my run the aliases field of every version came back empty, while the aliases existed. The listing compares three ways of asking the same question and, on the last line, shows the fields that a registered version carries with it:

import mlflow
from mlflow import MlflowClient

mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()
NAME = "churn-classifier"
print("search_model_versions:", [(v.version, list(v.aliases)) for v in client.search_model_versions(f"name='{NAME}'")])
print("get_registered_model :", dict(client.get_registered_model(NAME).aliases))
mv = client.get_model_version_by_alias(NAME, "champion")
print(mv.name, mv.version, mv.status, mv.source[:9])

# search_model_versions: [(2, []), (1, [])]
# get_registered_model : {'challenger': 2, 'champion': 1}
# churn-classifier 1 READY models:/m

The list from search_model_versions returns (2, []) and (1, []): no aliases. get_registered_model(...).aliases instead returns the correct dictionary, {'challenger': 2, 'champion': 1}, and get_model_version_by_alias finds the champion. If you build a "who is the champion?" check by taking it from the list, you may conclude that there is none. When you need the answer, ask for it with the function made for that question, and verify the check with a known case. I observed the behaviour with 3.18.0 and it may change with other versions.

The last line prints the logical name, the version number, the status (READY: the model is ready to use) and the beginning of the source, that is, the address of the artifact (models:/m-..., the model as an MLflow 3 entity). The run_id field of the same version is the lineage: from there you go back to the parameters, the metric, the dataset recorded as a tag. The registry does not copy the model: it catalogues it and manages its labels.

Two ways of loading the same model

The same model can be loaded in the native format, by version, or in the generic pyfunc format, MLflow's universal interface, by alias. They look equivalent and they are not:

import mlflow
import pandas as pd

mlflow.set_tracking_uri("sqlite:///mlflow.db")
X = pd.DataFrame({"tenure": [3, 60], "complaints": [4, 0]})
m1 = mlflow.sklearn.load_model("models:/churn-classifier/1")          # by version
m2 = mlflow.pyfunc.load_model("models:/churn-classifier@champion")    # by alias, generic format
print(m1.predict_proba(X)[:, 1].round(4))
print(m2.predict(X))

# [0.7993 0.1862]
# [1 0]

The first loads a precise version as a scikit-learn object, with all its methods, including predict_proba. The second exposes the same predict for every library. The trap is in the result: pyfunc.predict returned [1 0], that is, the predicted classes, not the probabilities. If the service expects a probability and loads the model in the generic format, it receives 0 and 1 and the business thresholds go wrong without any error. The remedy: load in the native format (mlflow.sklearn.load_model) when you need specific methods, and prove it with a test that checks the type and range of the output (see testing models with pytest).

A service that declares which version it serves

The principle "always record which version the service loaded" is achieved with a few lines in the startup code, in place of loading from a file. The registry address comes from an environment variable (environment variables and secrets), set here to sqlite:///mlflow.db:

from mlflow import MlflowClient
import mlflow, os

state = {}
NAME = "churn-classifier"
mlflow.set_tracking_uri(os.environ["MLFLOW_TRACKING_URI"])      # from an environment variable

version = MlflowClient().get_model_version_by_alias(NAME, "champion").version
model = mlflow.sklearn.load_model(f"models:/{NAME}/{version}")   # load by NUMBER: no race between the two lines
state["model"], state["model_version"] = model, version
print({"status": "ok", "model_version": state["model_version"]})

# con MLFLOW_TRACKING_URI=sqlite:///mlflow.db
# {'status': 'ok', 'model_version': 1}

The subtle detail is on the second-to-last line: the alias is resolved to a number and the model is loaded by number. If it were loaded twice by alias and between the two calls someone moved the alias, the declared number and the loaded model might not match. The version then ends up in the service's health check, described in the article on deployment. With this, "which model is answering?" has an answer you can read with a curl.

Comparing champion and candidate the right way

The promotion rule from before compared the auc tag of the two versions, and it hides a subtle error: those two values were measured on different splits (every training run has its own), and a difference of a few thousandths of AUC between different splits is within the noise. The honest comparison is between the two models on the same evaluation set, fixed and never used for training, called a holdout. The script retrains a candidate on new data (1,500 customers with another seed: imagine more recent data arriving), evaluates champion and candidate on the same holdout of 3,000 customers, registers the candidate and decides with the margin:

import mlflow
import numpy as np
from mlflow import MlflowClient
from sklearn.metrics import roc_auc_score

from churn.data import make_dataset
from churn.data import FEATURES
from churn.model import train

mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()
NAME = "churn-classifier"
MARGINE = 0.01

holdout = make_dataset(n=3000, seed=99)       # FIXED set, never used for training

def auc_su_holdout(model) -> float:
    return roc_auc_score(holdout["churn"], model.predict_proba(holdout[FEATURES])[:, 1])

champion_v = client.get_model_version_by_alias(NAME, "champion").version
champion = mlflow.sklearn.load_model(f"models:/{NAME}/{champion_v}")

with mlflow.start_run(run_name="retraining"):
    candidate, _ = train(make_dataset(n=1500, seed=7), seed=0, C=1.0)
    info = mlflow.sklearn.log_model(candidate, name="model")
    a_champ, a_cand = auc_su_holdout(champion), auc_su_holdout(candidate)
    mlflow.log_metrics({"holdout_auc_champion": a_champ, "holdout_auc_candidate": a_cand})
    mv = mlflow.register_model(info.model_uri, NAME)
    client.set_model_version_tag(NAME, mv.version, "holdout_auc", f"{a_cand:.4f}")
    if a_cand >= a_champ + MARGINE:
        client.set_registered_model_alias(NAME, "champion", mv.version)
        esito = "promoted"
    else:
        client.set_registered_model_alias(NAME, "challenger", mv.version)
        esito = f"NOT promoted (required margin {MARGINE}): v{mv.version} stays challenger"
print(f"champion v{champion_v}: {a_champ:.4f} | candidate: {a_cand:.4f} | difference {a_cand - a_champ:+.4f}")
print(esito)

# champion v1: 0.7208 | candidate: 0.7213 | difference +0.0005
# NOT promoted (required margin 0.01): v3 stays challenger

The champion is loaded by number, resolved from the alias. The two metrics end up in the run, the candidate in the registry with the holdout_auc tag, and it becomes champion only if it beats the champion by at least the margin; otherwise it stays challenger. Here the AUC is practically identical (+0.0005) and the candidate was not promoted. It is a useful result: the system retrained, registered, measured and left in service the model that was already there, with no human intervention and no risk. The candidate is version 3 and took the challenger alias, moving it from version 2.

Shadow mode: same requests, no risk

Even when two models have the same AUC they can decide differently on individual customers. Before promoting, a prudent habit is to run the candidate "in the shadow": it receives the same requests as the champion, but its answer is not used, only recorded for comparison. A miniature version, on 1,000 new customers, is the tail of the same script:

X = make_dataset(1000, seed=123)[FEATURES]
p1, p2 = champion.predict_proba(X)[:, 1], candidate.predict_proba(X)[:, 1]
print(round(float(np.abs(p1 - p2).mean()), 4))
print(int(((p1 >= .5) != (p2 >= .5)).sum()), "out of 1000")

# 0.0094
# 13 out of 1000

The mean difference between the two models' probabilities is 0.0094, and 13 customers out of 1,000 (1.3%) receive a different decision at the 0.5 threshold, even with practically identical AUC. The AUC measures the ability to rank customers; it does not say whether two models would make the same decisions. For a service whose predictions trigger an action (a retention phone call, a block), that 1.3% is a real change of behaviour even without a change of metric. In production you record the candidate's answers next to the champion's and analyse them after hours or days of real traffic.

There are two neighbouring strategies, often confused. The canary sends a small share of real requests (for example 5%) to the candidate, whose answers are used: more informative, but with a limited risk. The A/B test is an experimental comparison with random groups of users, to measure the effect on the business outcome; the article on A/B testing ML models covers it. The shadow is the safest: no user is subject to its answer.

Metadata, descriptions and aliases to remove

Every version can have a text description and free tags, and an alias that is no longer needed is deleted without touching the version. The script writes a description and a validation tag on version 1, then shows the aliases before and after deleting challenger:

import mlflow
from mlflow import MlflowClient
mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()
client.update_model_version("churn-classifier", "1",
    description="Logistic regression, synthetic data seed 0")
client.set_model_version_tag("churn-classifier", "1", "validated_by", "first-last")
print(client.get_model_version("churn-classifier", "1").description)
print(sorted(client.get_registered_model("churn-classifier").aliases))
client.delete_registered_model_alias("churn-classifier", "challenger")
print(sorted(client.get_registered_model("churn-classifier").aliases))

# Logistic regression, synthetic data seed 0
# ['challenger', 'champion']
# ['champion']

The description remains readable in the interface and from the API (get_model_version(...).description returns the string that was written). A minimal set of tags that spares you future questions: holdout_auc (the metric and on which set), dataset, commit, validated_by (in the listing above, validato_da in the Italian original) and the date. Versions can be deleted with delete_model_version, but doing so removes the possibility of going back to them and can break a service that loads them by number: it is an operation to perform only with a clear policy, for example "keep the version in production, the previous one and the last N".

The rollback drill: do it before you need it

A rollback that has not been tried is a hope. A ten-minute exercise, to do on a test environment:

1. Start with champion on version 1; start the service and note the answer to a fixed request (tenure=3, complaints=4: in my case 0.7993).
2. Move the alias to version 2 (set_registered_model_alias(... "champion", "2")) and restart the service. Check that the health check declares version 2 and that the answer to the same request has changed (in my case 0.8051).
3. Put the alias back on 1, restart, and check that the answer is back identical to the first.
4. Time the whole operation. If it takes more than a few minutes, write the exact commands in a runbook (the document with the release commands, discussed in the article on manual deploy and CI/CD), so that next time, perhaps at two in the morning, they are already ready.

Step 3 is the one that gives confidence: the answer "comes back identical" proves that the rollback really brings back the same behaviour, and that the model does not depend on something else that has changed in the meantime. A preprocessing file, for example, is a good reason to include the data preparation inside the saved model, as a scikit-learn Pipeline.

A typical case: the model always in the same file

Imagine a service that rebuilds a model every Monday (with a systemd timer, for example OnCalendar=Mon *-*-* 03:00:00 UTC) and always writes it to the same path, model.joblib, which the service reads from an environment variable. Re-reading it with the categories of this article: no numbered version, no alias, no place to record that day's metric, and a rollback possible only if a copy of the file exists. It is a choice of simplicity that is sustainable with one person and one model. The signals that say "now you need a registry" are precise: a retraining worsens a result and you do not know by how much; you want to compare two models in parallel; someone else must be able to decide the promotion. The step, when it comes, is small: save the model with log_model, register it, and have the service load models:/model-name@champion in place of the fixed path.

Four questions a registry can answer

A registry is not a place to "put models": it is what lets you answer, verifiably and in a few seconds, four questions that arrive in production sooner or later, often at the worst moment.

1. What is running right now? A version number, not "whatever is in the folder".
2. Where does it come from? The lineage: which run, with which parameters and data, from which commit.
3. What was there before, and how do I go back? The previous versions remain, and going back to one means moving a label.
4. Who decided, and why? Tags and descriptions on the version leave a trace that outlives the people.

If one of these questions, in your project, has the answer "it depends on who you ask", it is the sign that you need the registry, or at least its discipline. Whoever answers a user who reports an anomaly must know which model produced it; without a registry, going back means hoping that someone kept the right file.

Access and responsibility

Moving the champion alias changes the model in production: it should be an action with the seriousness of a code release. Three points to decide, whose answer depends on your installation and must be checked in its documentation, not assumed. Who can promote? If all team members have full write access to the registry, the promotion rule is only a convention: a shared registry requires a permission system or a single pipeline with the exclusive right to change aliases. Is there a trace of the changes? If your registry does not keep who moved the alias and when, keep it yourself, with a date-and-author tag or with a line in the release log. Do the services reload by themselves? If so, a move of the alias changes production without a release: convenient and dangerous. If not, a controlled restart is needed, the simplest and safest choice.

Trade-offs and limits

• Registry versus versioned folder: for one model and a single person, saving the files with the date in the name and keeping the last N may be enough. The registry pays off when you have to go back, compare, let another person decide, or serve more than one model.
• Loading by alias versus pinned version: the alias is convenient but changes behaviour without a release of the service; the version pinned in the code is more rigid but more traceable. With the alias, always record which version the service loaded.
• A registry to manage: you need a real database, backups and write permissions limited to those who can promote. The artifacts must be in a place the server can see: the database may say "version 3" while the file is in a folder that is invisible.

The common criterion is the same as for any process tool: you adopt the registry when the cost of not having it (unrecoverable models, promotions with no trace) exceeds the cost of managing it. Three limits do not depend on the choice: the registry does not decide for you (the promotion rule has to be written), it does not reload services that are already running, and it does not version the data (for that see datasets and models with DVC and lakeFS). Nor does it monitor the model in production: if quality degrades over time you need drift detection.

A complete retraining cycle

1. Trigger: a schedule (a weekly retraining, for example) or a signal of degradation.
2. Train a candidate, tracked as a run, with a reproducible environment.
3. Evaluate candidate and champion on the same fixed holdout, and check the behaviour tests too.
4. Register the candidate; promote only with the written rule, otherwise it stays challenger.
5. Release with a controlled restart and check the version being served.

This is, in miniature, what the series MLOps: from experiment to production calls the continuous training cycle. The module does not automate it: it gives you the pieces with which to decide, case by case, how much to automate, and how much automation makes sense is addressed by the article on manual deploy and CI/CD.

Try it yourself: four guided exercises

Exercise 1: register and load by alias

After tracking the four experiments, register the best one with register_model, give it the champion alias and load it with mlflow.sklearn.load_model("models:/churn-classifier@champion"). You should see Created version '1' of model 'churn-classifier' and a prediction for a test customer. Also try mlflow.pyfunc.load_model on the same address and compare the output of predict with that of predict_proba. What you learn: that the generic format returns the classes (in my test [1 0]) and not the probabilities.

Exercise 2: the rollback drill

Register a second version (with C = 10.0), give it the challenger alias, then move champion to version 2 with set_registered_model_alias. Reload by alias and note the answer for a fixed customer. Put champion back on version 1 and check that the answer comes back identical. You should see two different probabilities and then the first again (in my test, 0.7993, 0.8051 and again 0.7993 for tenure=3, complaints=4). What you learn: that the rollback is a change of label and that proving it takes a minute.

Exercise 3: the comparison on the same holdout

Write the script that retrains a candidate, evaluates it together with the champion on a fixed holdout, registers it and decides the promotion with a margin. You should see the two AUC values and the decision (in my test: 0.7208 against 0.7213, not promoted with margin 0.01). Change the candidate's data (another seed, more rows) and see whether the decision changes. What you learn: that the comparison has to be made on the same evaluation set, and that "do not promote" is a valid result.

Exercise 4: two models, different decisions

With champion and candidate, compute on 1,000 new customers the mean absolute difference between the probabilities and how many customers receive a different decision at the 0.5 threshold. You should see non-zero numbers even with nearly identical AUC (in my test, 0.0094 and 13 customers out of 1,000). Write in one sentence what it would mean for the service. What you learn: that the same metric does not imply the same behaviours, and the value of the shadow comparison.

For those who want to close the circle: make the service from the Dockerfile article load the model by alias at startup, expose the version in the health check and repeat the rollback drill through the service, restarting it. It is the exercise that joins almost the whole module.

What you can do now

You can register a model with MLflow, give it an alias and make a service load it by logical name. You can promote a candidate and go back by moving a label, with a written promotion rule and a margin compared with the noise, and you can compare champion and candidate on the same holdout and in the shadow. You also know two concrete traps: the list of versions can show the aliases empty, and the generic format returns the classes and not the probabilities. The next step is to take the model where it runs: deployment, where the model runs and how to choose.

The same series also includes ML-Ops: from notebook to service.